When we’re not creating the next big thing for photographers and collaborating with clients and photographers, we are busy searching the web for the best photographers around. What I’ve found while performing this search of photographer websites is that it’s time consuming.  Here’s why.

Photographer Websites: Why they are bad

First off, I should say that I understand that photographers want their websites to be creative and visually appealing, but it is a rare occasion when I find that a photographer has built in the idea that they’re actually in business.

Photographer Websites: Load Time

If it takes me 1-minute just to get the home page of a photographer website because I’m waiting for Flash to load, and upon reaching that home page I am presented with a nice image and an “Enter” button, then the chances are that I’m going to give up.  It is clear that you don’t have my time-interest in mind.

Photographer Websites: Contact Information

Every page of your website should have, at least, your email address, or a link to your email address, and, at best, your email address and phone number.  I don’t want to fill out a contact page; I don’t want to search your site for your phone number or email address. Make your page SCREAM “Contact me because I want your business”.

Photographer Websites: Too Many Photos

All it takes is about 20-images per photography genre portfolio for me to get the idea as to whether or not I like your style and think you will fit for a particular project.  If you have 100-images in each portfolio then you’re wasting my time by having my browser to download the thumbnail images for each of the 100-images.

Also, if you use an image hosting company to host your images, try not to make me visit your Flickr site to have me look at your photos.  There are many ways to display your images on your site without imposing great inconvenience to me.

Photographer Websites: Tips to Improve Yours

• Use Javascript, or HTML5, for your website.  Browsers are better equipped to handle either of these technologies than Flash.  Many times, with the various Javascript libraries available, you can mimic everything you can do for images in Flash, with Javascript.
• If you must use Flash, embed it so that at least the header of the page is still HTML.  Then, put your phone number and/or email address in this header and then I can copy and paste it without difficulty.
• Try to make your site load in under 10-seconds, because this is just enough time for me to click the link to your site, take a sip of my drink, and begin checking out your work.
• Keep it simple.  Simple doesn’t mean that you have to lose elegance.   Remember the reason that people are visiting your site is because they’re either interested in your work, or interested in hiring you.  Make sure that you add function to your form.

Introduction:

With all of the latest talk about SOPA and how many believe that it’s going to be the end of the internet as we know it, I was surprised to read that Google apparently doesn’t know how to detect copyright infringement.  I won’t say that it was easy coming up with the solution that I invented, but I will say that it seems that if Google wanted to figure out how to, say, transmute lead to gold then all they would have to do is put their minds to it.

The Method for Detecting Copyright Infringement:

The Control:

I view the detection of copyright infringement as a scientific test.  With this in mind, we set a reference, or control, and then every subsequent test results with a comparison to the control/reference.  The National Photographer Group has developed a system for detecting copyright infringement and has implemented a method to allow users to voluntarily create a reference for us to test against.  In other words, we authenticate their camera using our proprietary media authentication system and we register their camera with their account.

The Test:

Web applications can register to use our API and simply send an unmodified image or capture, complete with metadata, to our servers.  We perform an in-depth analysis on the media to attempt to determine who the media belongs to and then return a result as XML or JSON–user configurable.

The NPG Media Authentication System:

The NPG Media Authentication System can be implemented both remotely and locally depending on the demand the organization requires.  We offer affordable licensing of our technology, please contact customerservice@thenpg.com for an evaluation of your copyright infringement detection needs.

Recommendations:

• Do not allow users to upload media that does not contain metadata
• Promote the use of licensing standards, such as PLUS, to clearly define who is allowed to do what with the media
• Do not allow users to download content if they do not explicitly have permission to do so as spelled out in the PLUS license

The response from the release of the Lightroom® plugin has been tremendous!  I’m glad that people are finding it useful and the hosting service of value, too.  If you have yet to download our Adobe® Lightroom® publishing plugin with the QRcode label feature, then get it here

Corrections:

Our hosting provider made an oopsie and so some server configurations were changed so that the domain name for TheNPG.com did not properly resolve which caused the initial release of the Lightroom® plugin to fail.  Rather than change the server-side code, we thought it was best to standardize our URL and so the latest version of the plugin has this patch applied.

There have been other network issues that our network center has been working on.  These maintenance issues will hopefully be resolved soon, but please be aware that there may be intermittent times of a loss in service particularly in the early A.M hours.  While they are doing their thing, we are working on co-locating our services so that the newer versions of the plugin will be able to failover to another server that may be available.

Remember to stay up to date with our Twitter feed

-Lance.

My first job was as a DJ at an AM radio station.  It was one of those radio stations that is only on the air during daylight so every morning when the station goes on the air, and every evening when it goes off the air, the National Anthem is played.  I figured this would be the perfect way to kick off this round of service releases and so without further delay, I present, the National Anthem of the United States of America.

The American Dream

The American dream is one of prosperity and success.  This country is founded on the principles that if one works hard enough, and is persistent, then they will achieve whatever it is that they set out to achieve.  When I started the National Photographer Group, llc, I had a vision of a company that would help photographers become better at their business through developing technological tools and by facilitating connections between clients and the best photographers.  Today marks a very special day in history for me personally, and for the National Photographer Group, llc, in that from this foundation that has been set, we will grow; we will prosper; we will succeed; we will change the world.

New Services

Today is the first day of new services to be released.  There are a couple of other services that will be released over the next few days as we tie up a few loose ends.

Perhaps the biggest release is our new publishing plugin for Adobe® Lightroom®.  This plugin will enable a Lightroom® user to publish images to TheNPG.com servers–powered by Amazon servers.  The plugin will also allow photographers to create barcode labels for their images in an effort to make their re-print service more efficient and professional-looking.

This isn’t just any plugin; this plugin incorporates a camera-to-user-to-image authentication system invented and developed by National Photographer Group, llc.  In an effort to simplify the installation of this plugin, we offer the following How-to video:

NPG Lightroom® Download

How much does all of this cost?  Image hosting with TheNPG.com is a pay-for-what-you-use service.  Unlike other image hosting companies, you won’t be charged a large monthly fee if you’re only storing a small amount of data.   Instead you pay 25-cents per Gigabyte-month stored and 25-cents per Gigabyte transferred [both upload and download].  There is also a $1 monthly service fee to cover our processing of the payment.  An initial payment of $1.25 is required to begin using the NPG image hosting service.  You will be presented with the opportunity to pay when installing the plugin.

So what’s the catch?

Here’s the catch: Your account will be billed on a monthly basis.  Ok, that’s not much of a catch.  How about this: what if you are a photographer and can’t make a payment or two, will you delete the images?  Nope, along with being a pay-for-what-you-use-service, NPG Image Hosting is also a pay-what-you-can service so if you have a bill of $20 and can’t pay the entire $20, pay whatever you can afford and we will defer the balance forward to next month.  If a payment is not made in 60-days, your NPG hosting account will be suspended.  Failure to pay *something* in 90-days and the data is purged.  Still not much of a catch?  That’s because there is no catch.  Here’s how I see it:

When I was a photographer, I was the epitome of a struggling artist just trying to make it.  I realize that professional photographers aren’t always paid up front for their work, and sometimes it may take a few weeks to receive a check.  While I understand that not all photographers are in this situation, I am gearing our hosting service to all professional photographers.  If I can cover my expenses–redundant image hosting isn’t cheap–then I am OK with this payment system.

If you have any questions about the NPG Image Hosting service, or our publishing plugin, then please send an email to customerservice@thenpg.com and we will address your question.

Download Link

So be it:

TheNPG.com Lightroom Plugin Download

Enjoy!

Closing

In closing, I just want to thank everyone who has supported me over the past few years.  Also, this blog will be updated as our other services are rolled out this week.

– Lance.

*Adobe® and Lightroom® are registered trademarks of Adobe Systems Incorporated.  National Photographer Group, llc, is not affiliated with Adobe Systems Incorporated.

Here’s a sneak peek at the new offerings of TheNPG.com beginning November 5, 2011

RSS Media Streams

- At the National Photographer Group, our photographers photograph A LOT of corporate events.  One of the most popular requests from event organizers is to have a slideshow of the images so that the images can be displayed on a screen or other large video display.  Using an RSS media stream, the Audio/Visual folks can easily tap into the stream with their RSS enabled devices and overlay whatever music for the mood is required.  For the National Photographer Group photographer, creating the stream is simple and fully customizable.  Furthermore, because it is a RSS media stream, many devices such as WiFi enabled digital photo frames, Windows 7 and Macintosh computers, modern televisions and professional media displays can all access this stream.

Professional Image Hosting

- If you’re a professional photographer, then you want a simple interface; you want to be able to host many different image formats–and be able to display previews of the RAW formats; you want to be able to give your client access to the images you created for them; you want to be able to archive your images and be able to recall them quickly when a customer wishes to have an image reprint.  No other image host will do what we do at the price we will do it for.

Photo Shows

- You’re an event photographer and you want to be able to let event attendees to view the images from their event.  If they wish to print the images, you want their experience to be seamless and hassle-free.  Our Photo Shows are this, and will be more in the future.

Adobe® Lightroom® Publishing Plugin*

- What better way is there to upload images to our servers than through the same program that professional photographers use to process their digital images?  None.  This plugin creates seamless integration between Adobe® Lightroom® and TheNPG.com servers.  What’s more is that it integrates the very first user-to-camera-to-image authentication system invented by National Photographer Group.  This authentication system ensures that only the images that belong to a photographer, or are in public domain, are uploaded to TheNPG.com servers.  This plugin is under active development and will continue to evolve as the needs of photographers also evolve.

Barcoded Labels Printed From Lightroom

- This is a feature of our Lightroom® plugin.  Once the plugin is installed, a photographer can select the images from their catalog that they wish to print bar-coded labels for.  Then, with just a few clicks and selections, a PDF containing the information for each of those selected images in a printable format is created.  The photographer then prints out the labels and can adhere them to the reverse side of photo prints.  When a customer wants a reprint of a specific image, the photographer scans the QR-code on the rear of the print and Lightroom® searches the photographer’s catalog for that print.  Clever, huh?

Closing

These are just a few of the tools that we’ve been working on here at National Photographer Group.  TheNPG.com and the Lightroom® plugin are always under development as we find new ways for photographers to excel in their business.

We’re also working on new hardware solutions for event photographers that will bring a level of awe to their work.  Stay tuned.

If you have a suggestion for a hardware or software idea that you think will help the photography industry, please send your suggestion to us so that we can evaluate it and perhaps work with you to make it happen.  We will always give credit where it is due.

Until next time, happy shooting!

*Adobe® and Lightroom® are registered trademarks of Adobe Systems Incorporated.  National Photographer Group, llc, is not affiliated with Adobe Systems Incorporated.

It’s simple: a photographer sells prints to an event attendee and that event attendee comes back to the photographer with the print with the intention to have reproductions made.  Currently, there’s no quick way to identify a photo from an image in Lightroom®, until now.  A day worth of work, and we added this nice feature to our soon-to-be-released Lightroom® plugin.

How it works

Simply select the images that you wish to create labels for and then go to the File menu>Plugin Extras>NPG Label Maker.  A form is presented where you select the Avery® label size and up to four fields of meta data from the images, or from your account.  Click OK and your internet browser is opened where you are presented with a PDF form with properly spaced and formatted labels for the images that were selected.  There is a QRCode that contains the UUID identification number on each of the labels.  Simply load your printer with your Avery® labels, click print, and presto!

How to find photos from the QRcode?

We’re currently [as of Dec. 12, 2011] in the late stages of development of an in-Lightroom QRcode [Barcode] reader plugin.  This plugin will be part of version 1.1 of the NPG Publishing Plugin Suite and we plan to release it as a separate plugin for those who do not wish to use our publishing service.

What else is required?

An internet connection.  The Lightroom® API is simply not powerful enough to perform all that is required in order to produce these labels so we simply send the meta from the select photos, along with the label size selection, to our servers where we already have the framework to perform the complex operations in order to build these labels.
TheNPG.com Lightroom Plugin Download

Conclusion

There’s no more relying on memory and spending hours of searching through photos to find the exact one that a client wants reproductions of.  What’s more, because the UUID is used to search for the photo, Lightroom® will perform the search quickly so that you can move forward with ordering your prints.

We’re always looking for nifty projects like this to work on to improve the efficiency of  the photographer doing business.  If you suggest a new feature, and we implement it, then we will share the profits with you.  Please use our Suggestion Box to inquire further.

Cheers!

Lightroom® is a registered trademark of Adobe Systems Incorporated.  Avery® is a registered trademark of Avery Dennison Corporation. National Photographer Group, llc, is not affiliated with either Adobe Systems Incorporated, or Avery Dennison Corporation.

In July of this year I bought my first Android™ device.  I sort of knew what I was getting into because I had played around with the Android SDK a bit last year, but it wasn’t until I held the device in my hands that I began to realize it’s full potential.  I have since received the Nexus S phone by Google and have been completely fascinated by the two devices.  You know what this means: Homer has to build an application because it’s the cool thing to do.

Does it require batteries?

There are some thoughts being bounced around as to what this application might do.  Some have said that it would support one of our new products to be rolled out on November 5 of this year, and others have said that it would mimic the things that can be done on the website–at this time TheNPG.com isn’t very mobile-friendly, unfortunately.  Either way, the application will interface with our user API so perhaps it can do both.  Firstly though, it will support the new product that is to be rolled out on November 5.  My gosh I wish I could say what this will do because it’s just SO cool!

Do we need a gender changer?

Unfortunately, or fortunately, depending on how one looks at it, we will NOT be developing anything for Apple products, or any other mobile OS for that matter.

What would you want a mobile application to do?  Please leave your answer in our suggestion box.

I made an oopsie!

Homer, our lead programmer who looks a lot like a monkey, needed some help so we hired another to assist.  What happened on this person’s first day is that they accidentally sent out a mass email that was blank.  Or was it blank?  Apparently, there was a hidden message, and possibly some lottery numbers, but since you deleted the email, you’ll never know what it said.  At any rate, I apologize for this rookie’s mistake and let’s just be thankful that they don’t work at a nuclear reactor.

On the upside, I’ve decided to no longer use email as a mass communication method.  Instead, please follow us on Twitter  for the latest announcements and updates.

Until next time, keep your umbrella handy and the ice cream cold.

–Lance.

- National Photographer Group, llc is now a Co-worker-level member at Urban ReThink in Thornton Park, Orlando.

Urban ReThink is a first of its kind hub for creativity, collaboration, and transformation located in Orlando’s historic Thornton Park. By day, the space houses enterprising professionals in diverse fields from the arts to technology. By night and on weekends, we welcome the community with a “What will I discover today?” assortment of talks, workshops, and gatherings.

 

- New services!  There are some pretty nifty services to be launched on November 5, 2011–mark your calendars.  I would tell you what they are, but wouldn’t that ruin the surprise? Ok, here’s a hint:

• There are 5-things, some old, some revolutionary to the photography industry
• You may want to re-consider your image hosting service
• Wouldn’t it be cool if you could…

I’m trying to decide if I’ll have a formal launching party–something that I have yet to do for any of the new service launches thus far.  I’m leaning toward a “Yes” on the launching party because the services to be announced are simply worthy of such a party.

- A-List! As one might imagine this is a V.I.P [Very Important Photographer] list.  As you work with us and we develop a positive professional relationship–as noted more by feedback from our clients and your work than from any personal opinion–we will begin to add you to this exclusive A-List.  You cannot pay to get on the A-List, and slipping me a $20 at the launch party isn’t going to influence me, but if you continue to provide exceptional service to our clients then you will be added to the A-List.

Being on the A-List means:

- The clients who pay to access the A-list will see your contact information above B-list photographers

- You will be given precedence by our project managers when estimating a project

- You will maintain this A-List status for as long as you are a member of the site and you maintain your exceptional client feedback and work.

- Suggestions Galore! I’ve put together a very simple suggestion form at a special link on the website.  Only those who read this blog can access this link because it is not in the navigation on the Home page. Go Here to submit your suggestion.  Essentially what has happened is that I’ve run out of great ideas to develop and I’m at a point where I like where the site is–after the launch of these new services, that is. Here are some general subjects that I’m looking for:

• New online software tools/features.  Go nuts!  I can create anything!
• Is there a hardware device that you wish you had in your gear bag? I’m not talking about a new lens, or a new camera; rather, I’m talking about devices such as high-speed triggers, multi-camera synchronizers and things like that, but are not that.
• A new concept for monetizing on your images.  Anything is possible; impossible is nothing.

I want to be your R&D team.  Give me suggestions on things that will help you manage your business, or perhaps a new product that you’d like to offer your clients.  I’m not kidding when I say that I can do anything.

- Merging!

Ok, I didn’t mean to scare you.  There is no company merger happening now or in the future, although I might entertain something like this if it means I could continue working on the company without needing my full-time laser job.

Anyway, what I mean by “merging” is that when I first started the National Photographer Group, llc, the vision that I had for the company is not the same vision that I have for the company now.  A few months ago, I asked the photographers for suggestions on improving the service and a couple of you said that you would rather have a lead that you knew you were going to get than to buy a cheap lead that you may not get.  This makes sense, and it is actually something an advisor told me early on, but I didn’t want to listen, until now.

What has happened is that I’ve removed the function to search for projects and here’s why:  Instead of buying leads, that may or may not work for you, I’m now working directly with the client to understand their photography needs and I, or another project manager, will contact you [the photographers] for an estimate.  This means that there’s a bit more competition for the photographers, but if you are on the A-List, then you will be given precedence during these estimate wars, if you will.

- Philanthropy!

Before becoming a professional photographer, I worked with the Library of Congress in putting books and magazines onto audio cassette tapes in a specialized format for the blind and physically handicapped.  Doing this work, I learned how important it is for those, particularly without sight, to have programs that will advance technology as it applies to blindness.  With this in mind, I am pledging to give at least 10% of the profits from National Photographer Group, llc to the American Foundation for the Blind. AFB is a great cause and I think giving to them brings our mission full circle.

Closing:

In closing, I want to thank everyone for their continued support of National Photographer Group, llc and TheNPG.com.  Remember to mark your calendars for November 5, 2011 and those photographers in Orlando will receive a special invitation to the launch party as the date approaches.

If you have any comments, please leave them below.  If you would like to inquire about more information about National Photographer Group, llc, please contact us.  Thank you!

Over the years of developing TheNPG.com, I have had to learn about new technologies that I couldn’t imagine when I first started this project.  In fact, when I started coding the first version of TheNPG.com, I had no idea how to code Javascript and my knowledge of PHP was very basic, which meant my code was not efficient.

Efficiency is the key when programming a web service for medium-to-large-scale usage; often times the technology that we are most familiar with becomes the screwdriver when we really need a hammer, if you will.  However, if we take a moment to step back, invest a little time and research, we may find a better solution and we may just learn something in the long run.

Today, TheNPG.com, while it still has it’s drawbacks, is pretty darn efficient for as large as the script as a whole is.  Over 2MB of code make up all of the classes and external libraries TheNPG.com uses to perform it’s functions, that’s quite a bit, but thankfully it is not all run at once.

I rarely write about the back-end technology of TheNPG.com partially because a lot of the technology is proprietary and a core part of our business, but the past few days have unearthed some knowledge that I would like to pass on to others who may be in a similar situation.

Overview:

PHP is full of built-in functions that can perform just about anything a programmer would want to do, but often times there is a better way.  Case in point: file downloads.  It sounds easy easy enough: there’s a file on a server, somebody wants to download it, let them download it.  But, in reality, it is far more complex especially when factoring in efficiency and different technologies.  TheNPG.com uses Amazon’s S3 service to store files and there is the option to use Amazon’s Cloudfront technology to serve the files as a content delivery network.  It really is a fantastic service and it works well as a file hosting service, if you know the ins and outs of getting the files where they need to be, and making sure that the people who are downloading those files are allowed to do so.  We could have PHP do the bulk of the work by verifying privileges, fetching the file and serving it to the browser, but when we consider that PHP loads the entire file into memory, our efficiency is reduced and we have the potential of bringing our server to a stand-still.

The Objective:

The objective to our download service is to provide quick access to file downloads, which can be very large, while reducing the costs associated with transferring files between servers.  You see, Amazon S3 charges not only for storage of files, but also for requests to put and get the file, and the bandwidth required to move the file; it adds up!

Why not just host the files on our web server that has unlimited storage and bandwidth?

This was a question that the answer took a while for me to understand.  To better understand the answer, we have to consider the two different types of servers: the web server and the file server.  The web server is busy serving pages and dynamic content, it is not set up for redundancy, in other words, the disk could fail, but since the web server only has files on it that are required to generate a web page, it’s not a big deal because these files are generally small and easy to backup.  A file server, on the other hand, is built for redundancy; it might have several disk drives of a seemingly unlimited capacity where when one, two, or even three drives fail, there is still enough data on the others that the files can be recovered.  A file server is like a rock and a web server is like a race car; sometimes the race car crashes, but the rock will always remain in tact.  So, while using a web server as a web server/ file server is cheap, easy and really fast, it doesn’t make sense when you’re dealing with other people’s prized files.  For this reason, we have integrated Amazon’s S3 file service into TheNPG.com.

So, how do you connect the two services efficiently while keeping costs down?

Simple [of course I would say that since I've figured it out ].  For each file that is uploaded to TheNPG.com servers, we create different sized images, if the file is an image, and then put them all on Amazon’s servers using a PHP class for S3.  During this upload, we set special request headers that, when called via HTTP, will cause the browser to open a dialog box asking the user to either open the file, or save the file.  Because our PHP class accepts an array of request headers as an argument, our code looks like this:

$request_headers = array(
"Content-Type" => $filetype,
"Content-Disposition" => "attachment; filename=".$filename);

This sets two headers that will be sent to a browser if the file is called via a HTTP request.

But why?

TheNPG.com uses a caching system which works like this:

1. User clicks on link to download the original file
2. PHP checks the cache to see if the file has been downloaded in the past 24-hours, or so
3. If the file has not been cached, we do 2-things simultaneously: we send a header to the browser that redirects to the CDN server, and upon the browser connecting with the CDN server, the aforementioned request headers are output which causes the browser to bring up the download dialog.  While this is happening, we use the following code to fetch the file for our cache:

exec("wget -q -nc $location -O $store > /dev/null 2>&1 &");

What the code above does is it executes a shell program “wget” with the options to be quiet [-q], and to not download the file if it already exists in the directory [-nc], then we pass the cleaned location of where the file is, followed by the -O option and the location we wish to store the file.  The next bit of code is the real gem as it allows the program to continue while freeing itself from the bounds of the PHP script that called it.  In other words, we’re really doing tw0 things at once!

Why use exec and wget instead of readfile()?

Readfile works really well for smaller files in PHP; however, if we were to use readfile for larger files, say 25MB, then we would have to wait for the file to be downloaded by PHP before allowing anything else to happen.  If we can quickly send special headers to allow the user to immediately begin downloading their file, then they don’t have to wait for the two servers to exchange communication before letting the user download the file.  Also, wget has many neat features built into it that give it advantage over any PHP function, including the -nc option.  Why use PHP to do such heavy lifting when we can use a program that is built for this purpose?

So, once the file is cached on the local web server, we have the same issue with deciding how to serve the file.  Do we let PHP do it since the file is local now?  Yes and no.  For small files, it’s ok to let PHP handle them, but for the larger files there’s another gem called mod_xsendfile.  This is an Apache module that is useful for handling larger file downloads.  Essentially, when this module is installed and enabled, the webmaster simply sets the XSendfile header and the mod_xsendfile module does the rest without tying up PHP.  Here is an example of how to set the header:

header("Content-Disposition: attachment; filename=\"".$filename."\"");
header("Content-Type: ".$filetype;
header("Content-Transfer-Encoding: binary");
header("Content-Length: ".filesize($filepath));
header("X-Sendfile: ".$filepath);

And with that, PHP is free and Apache takes over with handling the local file serving to the user.  It’s just as quick as the first download request and just as transparent to the user.  If the file is popular, there can be significant savings in bandwidth transfer and request charges when using this method.  Of course, requesting the file twice means that there is a risk that if the file isn’t requested a subsequent time from the first then there is a bit of waste, but that’s ok because in the end, the savings outweighs the waste.

To keep the server clean, and the cache fresh, we setup a cronjob to run a script that checks to see if any of the files in the cache have been stagnant for the given amount of time.  We maintain a fetch-count along with a “last-requested” time stamp that allows us to maintain the cache so that our web host stays happy.

Also, with keeping in the quick delivery of files to the user, it was debated whether or not to use memcache, or a similar program, in order to maintain the cache index–or even the files themselves if they are small enough.  Ultimately, the decision was made to not use memcache at this time because it is something else that we need to install and maintain when we already have the ability to use MySQL memory [HEAP] tables without much extra effort, but with similar fetch speeds.  Our memory table simply keeps track of the file location, the original file name, the file MIME type, the fetch-count, time created and last accessed along with a hash index.  Keeping the table small, and using queries that use the index, the fetch time from MySQL is very quick, 0.000011 seconds, on average.

Conclusion:

There’s a lot of information here, and probably a lot more rambling seeing that it is pretty late at night, but hopefully this information is helpful to someone else wondering how best to manage file downloads.

In short:

• Use a local cache, but keep it clean and down to only the records which are required
• Get files for the cache using exec and wget -q -rc behind the scenes
• When files are put onto S3 servers, set the request headers to cause a download box to display when the download is clicked, then redirect the user to the S3 server where the file is downloaded directly from that server, use DNS to cloak the location of the file so the user cannot know where the file is hosted, exactly, OR use time-sensitive pre-signed URLS and only allow authenticated access to the files hosted on S3.
• Use MySQL memory [HEAP] tables, or memcache, to maintain a catalog of the cache
• Use mod_xsendfile to serve local files without tying up PHP and a lot of memory

EDIT:

I have read in other forums/blogs the dislike of using mod_xsendfile because, so it is claimed, the programmer is letting go of control of the file so other things cannot be done, such as database logging.  This is not true if you use PHP as FastCGI; the FastCGi processes do not end once the file is passed to the user; rather, they continue to process until the script completes.  This method is superior to other methods, and that’s the bottom line.

If you have any ideas that you would like to contribute in regard to the ongoing development of TheNPG.com then please send them to CustomerService@TheNPG.com.  Each and every email is read and will be replied to.  You can think of the National Photographer Group, llc as your personal web application developer without the high price.