The Niklas Lindblad Organisation


 

PLEASE NOTE THAT THIS IS AN ARCHIVED 2007 VERSION OF THE ORIGINAL SITE.

Code monkeys are monkeys too

"Ripping" original, copyrighted music off the internet in order to convert it to formats like MP3,MP4,AAC,3G etc is an ongoing problem for the music industry. Although a YouTube spokesman said: "Our terms of service prohibit the downloading, or copying of videos on YouTube without explicit consent from the copyright holder" it doesn't seem to stop most folks from doing it. Is ripping software legal? In the abstract, ripping software is not illegal, says American intellectual property lawyer Thomas D Sydnor II. YouTube won't do anything about YouTube ripping unless the content owners have explicitly stated that a user is not permitted to download their origianl copyrighted material from the site.

And here is another site helping those who want a Youtube command-line ripper. No one seems in the least bit concerned. Opps!

******

Youtube Ripping: Reloaded

The initial version of the Youtube command-line ripper appeared on Digg without ever reaching any sort of maturity. There was at most an hour between the beginning of the coding process and the massive public exposure. I am grateful for all the feedback I have received and, as promised I now announce a total rewrite in PHP that is confirmed working on Linux, Windows and OS X (just about any platform with PHP actually).

You've gotta admit, we are fast and would have even been faster if my second in command had been around. However, he is recovering from surgery to remove an IVC filter that had been implanted 6 months ago to trap blood clots due to his condition called deep vein thrombosis (DVT). J was unable to take blood thinners so the IVC filter was recommended.  When Jay learned that the FDA states that most IVC devices should be removed between the 29th and 54th day after implantation or as soon as the danger from blood clots has passed, he contacted his doctor. Well, it turns out there were complications removing the filter, fortuantely not dibilitating or fatal. However, Jay did look into hiring an IVC filter lawyer, just in case. Apparently there are a number of lawsuits either settled or pending regarding these medical devices, so Jay consulting with a lawyer was not crazy. When you do the research online for INC filter / lawsuits, the information regarding the behavior of some of these manufacturers is downright shocking. ANyway, Jay is OK but still recovering. Can't wait till he's back.

Version 0.40 is available for download.

PHP5 is available for Windows as binaries from the PHP.net site
Most Linux distributions have a package in their repositories
There is a port of PHP5 available for OS X users

******

10 Responses to “Youtube Ripping: Reloaded”

Youtube Ripper: Collector’s Edition « Niklas Lindblad Says:
May 20th, 2007 at 2:39 pm

[…] For a working script, see this post […]

Kz Says:
June 8th, 2007 at 2:32 pm

Hi there,

I have downloaded the script but cant seem to open the readme or todo files; perhapes you should make this into notepad or something everyone has…….

nlindblad Says:
June 8th, 2007 at 7:41 pm

The files are in plain-text. Rename them to README.txt and TODO.txt if Notepad can’t open them.

******

James Attard Says:
June 19th, 2007 at 2:56 pm

There was a slight bug in the script - as of today at least, it didn’t work:

replace line 40 with:
ereg(’/player2.swf\?hl=en&video_id=.*’,$match,$id);

and line 42 with:
$unique_id = preg_replace(”/\/player2.swf\?hl=en&video_id=/”,”,$str);

I have one question though - although I loved your script I still cannot understand how did you arrive that youtube calls videos through the http://www.youtube.com/get_video - i didn’t manage to find that string with wget. Did you use some sniffer to arrive to that conclusion??

Thanks !

******

Theo Says:
July 15th, 2007 at 8:23 pm

Great script, however as of today as least the page switching bit doesn’t work (it will only download from page one). To fix:

replace line 53 with:
$matches = preg_grep(”/&p=r&page=[0-9]*/”,$html);

 ******

Theo Says:
July 15th, 2007 at 8:25 pm

Hmmm, not sure how to escape special commands in Wordpress so the code above won’t work. Basically you need to replace the &’s with “& amp;” (omitting the spaces)

******

Paul Says:
July 21st, 2007 at 2:51 pm

Try &
Paul Says:
July 21st, 2007 at 2:53 pm

Err.. & :S

******

school teacher Says:
August 17th, 2007 at 12:18 am

wartornhols@yahoo.co.uk

I’m only a thick drama teacher who can’t understand scripts! not on the computer anyway. I need to rip a piece from youtube to shiow my kids, as youtube is banned in schools. I need the following youtube vid as any video file to embed elsewhere.

http://www.youtube.com/watch?v=pjd-xkR9mxw

could anyone rip this for me and email me the result?
cheers.
I’m trying to learn

******

Raphael diSanto Says:
August 25th, 2007 at 3:05 am

YouTube just changed the format of their HTML in their individual video pages. You can’t search for swfobject anymore to find the unique video ID unless you can fool the javascript into thinking you have a valid Flash player. However, the unique video ID can be found by regexp-ing for /watch_fullscreen.+?”/

******

Youtube Ripper: Collector’s Edition

Unfortunately, this script is broken now. Youtube changed things internally and the script broke. For a working script, see this post

Today I had an interesting conversation with my fellow blogger themak and he asked if it would be possible to download a bunch of videos that he knew followed a regular naming scheme and that were uploaded by two different users. After some coding I came up with a script that will download all videos uploaded by a specific Youtube user where the title matches a pattern (regexp compatible). The final output is flash videos named after the unique video ID.

$ sh youtube-user.sh rpoland "Andy Mckee"

Would download all videos with Andy Mckee in the title listed on the user rpoland’s profile page.

Update: Mike wrote a PHP port!

Update again: Regarding the Digg comments, the script now works with all profile pages and I re-wrote it to feature functions instead of one big one-liner.

Hopefully last update: Fixed the last regexp flaws and removed a flag to xargs that stopped it from running on OS X. Also added a dry-run mode.

************

122 Responses to “Youtube Ripper: Collector’s Edition”

I’m Mike » Blog Archive » YouTube Video Ripper in PHP Says:
April 8th, 2007 at 6:00 pm

[…] It doesn’t have a progress bar (since it’s not using wget), but it gets the job done.read more | digg story Save This […]

******

x_rob Says:
April 8th, 2007 at 7:39 pm

W000t! You’ve been dugg! Front page!

******

technology blog Says:
April 8th, 2007 at 8:18 pm

how can .ph be used on web ?
thanks… great post…

******

Lasse Havelund Says:
April 8th, 2007 at 8:20 pm

Hehe, great work on the script–and congrats on the Digg frontpager :D

******

majeed Says:
April 8th, 2007 at 8:34 pm

check out this ripper
http://www.tubularapp.com/blog/
good luck

******

Jay Says:
April 8th, 2007 at 9:07 pm

there is no -r option for xargs in Intel Mac Book. I removed it and also tried replacing it with -t but I get a http 303 See Other. error. I am using the same example as give above for test case.

******

Ryan Says:
April 8th, 2007 at 9:18 pm

tubesock is a better ripper

******

nlindblad Says:
April 8th, 2007 at 9:18 pm

@Jay: The -r option is a GNU extension and it only adds some character checking.

The 303 is caused by the script at http://youtube.com/get_video which handles load balancing and redirects you to the actual content on a sub-server like http://sjc-v88.sjc.youtube.com/.

******

nlindblad Says:
April 8th, 2007 at 9:27 pm

@Ryan: You are aware about the fact that you’re comparing a *SCRIPT* to a full-scale application?

Thousands of lines of code in a proper programming language compared to 39 lines of text-processing commands put together to parse some links.

Hours of planned development and testing compared to a few minutes of script writing.

Very mature comparisons I must say…

******

YouTube Video Ripper « News Coctail Says:
April 8th, 2007 at 9:29 pm

[…] Video Ripper Filed under: Uncategorized — recar @ 8:28 pm YouTube Video Ripper A script that will download all the videos uploaded by a specific YouTube user where the title […]

******

themak Says:
April 8th, 2007 at 10:42 pm

Thanks for the script. Props for the frontpage of digg. :D

******

Thanks to Niklas at themak.org Says:
April 8th, 2007 at 11:35 pm

[…] After an interesting chat with Niklas regarding my desire to learn the guitar he coded my a nice shell script to download me some youtube videos. […]

******

video clips » Ripper: Edition Youtube Collector s Says:
April 9th, 2007 at 3:17 am

[…] The final output is flash videos named in the back of the unique video ID. … After some coding I dawned up with a script that will download the whole thing videos uploaded by a specific Youtube user where the title matches a pattern (regexp well suited). read more… […]

******

music video » Ripper: Edition Youtube Collector s Says:
April 9th, 2007 at 3:29 am

[…] The final output is flash videos named in the back of the unique video ID. … After some coding I dawned up with a script that will download the whole thing videos uploaded by a specific Youtube user where the title matches a pattern (regexp well suited). read on… […]

******

video codes » Ripper: Edition Youtube Collector s Says:
April 9th, 2007 at 3:33 am

[…] The final output is flash videos named in the back of the unique video ID. … After some coding I dawned up with a script that will download the whole thing videos uploaded by a specific Youtube user where the title matches a pattern (regexp well suited). continue here… […]

******

video ipod » Blog Archive » Edition Youtube Collector s Ripper: Says:
April 9th, 2007 at 3:39 am

[…] persist reading Rooney, Carvey plus Lane - VIDEO … The final output is flash videos named subsequent to the unique video ID. In this, the second of two how-to galleries, you’ll learn how to open your overlay in QuickTime Pro also merge it with your video to fashion a translucent worm in the spot of your movie. After some coding I arrived up with a script that will download the whole lot videos uploaded by a specific Youtube user where the title matches a pattern (regexp similar temperament). troth aware: you do need to upgrade QuickTime to … … college-occasion grandson (played by Dana Carvey) also his roommate, played by Nathan Lane. Meg Ryan similarly starred in the series, which, per IMDb, ran for a whopping eight episodes. … more… […]

******

fiona apple video codes » Overlaying Pro HOWTO: QuickTime video in a Says:
April 9th, 2007 at 3:40 am

[…] videos uploaded by a specific Youtube user where the title matches a pattern (regexp well suited). read rest of article… […]

******

funny cat videos » How Videos? I Do Download YouTube Says:
April 9th, 2007 at 3:54 am

[…] persist reading Rooney, Carvey plus Lane - VIDEO … The final output is flash videos named subsequent to the unique video ID. In this, the second of two how-to galleries, you’ll learn how to open your overlay in QuickTime Pro also merge it with your video to fashion a translucent worm in the spot of your movie. After some coding I arrived up with a script that will download the whole lot videos uploaded by a specific Youtube user where the title matches a pattern (regexp similar temperament). troth aware: you do need to upgrade QuickTime to … … college-occasion grandson (played by Dana Carvey) also his roommate, played by Nathan Lane. Meg Ryan similarly starred in the series, which, per IMDb, ran for a whopping eight episodes. … more… […]

******

video camera » Edition Youtube Collector s Ripper: Says:
April 9th, 2007 at 4:01 am

[…] persist reading Rooney, Carvey plus Lane - VIDEO … The final output is flash videos named subsequent to the unique video ID. In this, the second of two how-to galleries, you’ll learn how to open your overlay in QuickTime Pro also merge it with your video to fashion a translucent worm in the spot of your movie. After some coding I arrived up with a script that will download the whole lot videos uploaded by a specific Youtube user where the title matches a pattern (regexp similar temperament). troth aware: you do need to upgrade QuickTime to … … college-occasion grandson (played by Dana Carvey) also his roommate, played by Nathan Lane. Meg Ryan similarly starred in the series, which, per IMDb, ran for a whopping eight episodes. … continue… […]

******

dan Says:
April 9th, 2007 at 5:31 am

CastCluster does this and has a UI and makes RSS feeds from the results… and converts the videos to H.264
http://castcluster.blogspot.com/index.html

******

Live Television Says:
April 9th, 2007 at 1:45 pm

quite nice!

YouTube Video Ripper at The p2p Blog Says:
April 9th, 2007 at 2:49 pm

[…] Check it out […]

******

Andy Says:
April 9th, 2007 at 3:11 pm

Why use a ripper? Just go to the Internet Temporary Files folder, find the YouTube file there (you can guess by the size and date, although there will be no extension), copy it and paste it to another location on your hard drive and rename it to something meaningful with .avi extension.

******

mike Says:
April 9th, 2007 at 4:06 pm

Can you make the script rename the file with the string contained in the page’s title? I mean, the video is on a page. The page has a title. Make the video been named after the page it is in, or we will end with lots of videos like kj1234ku2.flv, 123kjhj1234h.flv, etc.

******

nlindblad Says:
April 9th, 2007 at 4:14 pm

The initial version featured some code that extracted the title as well.

I updated the previous script.

Thanks for the suggestion.

******

mike Says:
April 9th, 2007 at 8:18 pm

YESSSSSSSSSSSSSSS!!!! NOW IT IS PERFECT!
the only thing I had to do is to remove the “-r” option on my Mac OS X.
it does not work with this option.
thanks!!!!!!

******

Josh Says:
April 9th, 2007 at 8:52 pm

Hi, I don’t know if it is a problem in the script or on you tube. I’m using a Mac and I have removed the “-r” option.

this guy named melah65 has 5 videos that I want to grab from youtube.
the videos are named

UFO Files: Black Box UFO Secrets Revealed ( part 1 of 5 )
UFO Files: Black Box UFO Secrets Revealed ( part 2 of 5 )
UFO Files: Black Box UFO Secrets Revealed ( part 3 of 5 )
UFO Files: Black Box UFO Secrets Revealed ( part 4 of 5 )
UFO Files: Black Box UFO Secrets Revealed ( part 5 of 5 )

I tried the following command

$ sh youtube-user.sh melah65 “UFO Files: Black Box”

after grabing part 1, 2 and 3, the script stopped.

I have tried with other videos from other guys, the problem is the same. Not all videos are grabbed. After some videos, the script stop and no error is shown.

******

Josh Says:
April 9th, 2007 at 9:05 pm

I discover part of the problem. The guy named parts 1, 2 and 3 using “:” after the word “Files” and parts 4 and 5 using a dash after the word “Files”… anyway as the script uses regex it should not be a problem…

******

nlindblad Says:
April 9th, 2007 at 10:51 pm

@Josh: I’ve done some serious re-writing with the regexp in the crawl() function.

******

Josh Says:
April 10th, 2007 at 2:21 am

Niklas, now it is working fine!!!!!!!!! You’re the man!

******

MacBros Says:
April 10th, 2007 at 5:08 am

I just use the plugin that you can get for Firefox.
Neat concept though. If I ever find the need to copy a users complete video collection.
Shudders at the thought of a Numa collection.

******

MonkeyMan Says:
April 11th, 2007 at 4:21 am

Sorry, but I will have to say that the script is not working 100%. I would say that it is working 99%.

Using the test command as in
$ sh youtube-user.sh HeadUnit420 “BBC” test

I see this guy has the following videos

* C4cXGJFGeJ0 as BBC Space - E2P1 HD.flv
* Jn_sPcXYSVY as BBC - Space E1/P2 HD.flv
* aV4ADdHz1no as BBC - Space E1/P1 HD.flv
* iyftv9IGBCo as BBC - Space E2P2 HD.flv
* nN4wUbR-ujI as BBC Space - E1P3 HD.flv
* uGyIUwK-H7M as BBC - Space E2P3.flv

“BBC” is the string all videos have in common. Use this and those videos with “/” in the name will not be grabbed.

Anyway, thanks for your efforts.

******

nlindblad Says:
April 11th, 2007 at 1:57 pm

@MonkeyMan: For a very good reason. It would not be convenient with a filename containing ‘/’ on a UNIX system . :)

To fix this problem the filename that gets based on the video title should be properly escaped using another regexp replacing or removing things like slashes and other special characters.

Thanks for letting me know about this issue that I hadn’t considered before.

******

PJ Says:
April 18th, 2007 at 3:08 pm

HOW do this actually work? WHAT do I need to use this script thing?

******

nlindblad Says:
April 18th, 2007 at 5:05 pm

@PJ: It uses regular expressions on the Youtube pages. You need Bash and wget, which means it will work perfect on most Linux distributions, *BSD and OS X if you install wget.

Possibly might work with Cygwin as well.

NLindblad.org