It's a Doozy!
http://whencanireusethiscalendar.com/
Now you can go digging through that chest of crap from the 1990s and pull out your favorite cute puppies calendar. In 2010, you can re-use calendars from 1999, 1993, 1982, 1971, 1965, 1954, 1943, and 1937.
This morning I received a call from a gent with a Boston accent. He indicated that he represents a firm that is displeased with some data I’m using on isnoop.net. According to the caller, my theater listings page is using his client’s intellectual property and I’m not properly licensed to do so. The lawyer seemed nice enough. Perhaps I should have kept him on the phone longer so he could tick up some more billable hours…
Like some other things I’ve developed, theater listings was a simple service I wrote for myself to clean up an otherwise cluttered interface and make the data available in my favorite feed reader. Over the years, many people have written with questions and thanks regarding the page. Thank you to everyone who used the service. I hope you might find some of my other tools just as useful.
As of now, the theater listings page is closed. If you still want this information in your web browser, check out Google’s movie listings service. For you feed reader junkies, Yahoo Pipes is widely known as a useful service for turning any web page into an RSS feed.
I’ll investigate the possibility of re-sourcing the data, but don’t get your hopes up. Also, for those who are already firing up their email clients to ask me for the source code, hold your horses. I’ve been working up a post on ethical screen scraping and now I can finally share it without being hypocritical. I won’t share the source, but look forward to an interesting and useful guide to capturing and reusing data on the web, including some advice that should help prevent you from getting your own C&D.
As with most of my web toys, FeedSifter.com started off as a tiny tool that served a very simple need I had. Assuming a handful of people might have the same need, I publish most of these utilities and some of them actually manage to become fairly popular.
FeedSifter is a simple service that allows you to filter an RSS or ATOM feed for various keywords. There are many other services out there that do this same thing, but this site is anonymous, uncluttered, and intuitive–exactly what I wanted at the time.
Looking at the traffic stats today, I’ve found that feedsifter.com managed to become fairly popular while nobody was looking. Over the past 8 months, daily traffic has been steadily increasing and it is fast approaching 2,000 requests per hour. That’s a pleasant surprise and a good indication that I should put some effort into finishing those final few features I never got around to implementing years ago.
As a follow-up to my last post, here are a few tips to help keep you from driving your site users away with misguided password restrictions.
Your tweets may be precious to you, but as a web developer, you should understand the differences between password security for Twitter and for online banking. Consider the monetary and legal damages that to both you and your customers if their account were compromised and plan accordingly.
Since you are properly protecting passwords by only storing a hash, there is no reason to limit characters they use in their passwords. Don’t tell your users they can’t use symbols or start their password with a space if that’s what they want to do. In the end, all you’ll be storing is an alphanumeric hash of the password anyway, so it shouldn’t matter if they send you a bunch of binary gook.
Set reasonable requirements for password rotation. If yours is a high-security system, it is reasonable to require password rotation every few months. In order to prevent abuse, it is even reasonable to check for duplicates against the last N password hashes or all of the hashes used the last few months. Before implementing any password rotation scheme, be sure you’ve visited and revisited #1 above.
Despite the practices of some sites, there is rarely if ever a good reason to keep an unlimited log of every password a person has used, never letting them be reused.
Let’s assume someone really wants into one of your user’s accounts. If they had the capabilities of attempting a blistering 1,000 different passwords per second, here’s how long it would take to try every possible combination:
| Min length | a-z | a-z, A-Z | a-z, A-Z, 0-9 | a-z, A-Z, 0-9, Symbols |
|---|---|---|---|---|
| 6 | 3 days | 225 days | 2 years | 22 years |
| 7 | 90 days | 32 years | 110 years | 2,035 years |
| 8 | 6 years | 1,663 years | 6,814 years | 191,258 years |
| 9 | 166 years | 86 millennia | 422 millennia | 17,978 millennia |
The exponential growth of possible combinations makes password cracking infeasible pretty quickly. Assuming 1,000 attempts per second is ludicrous against a webserver, but assuming your database isn’t compromised, this should be the only means an attacker can brute force an account. The time to compute reaches absolute absurdity after just 8 lowercase characters.
As in character taboos, what do you care if your users have 100 character long passwords? It is reasonable to put a ceiling on password lengths to prevent blatant abuse (perhaps in the neighborhood of 500-1,000 chars), but don’t limit your users to a 12 character password because that’s all your schema can hold.
Remember that you should be hashing passwords and that most hashes produce a fixed-length output no matter the input.
Just a quick note to any developer, site owner, or project manager who is in charge of developing a user login system:
Don’t put unreasonable restrictions on usernames.
It is sensible to prevent people from creating names containing certain characters or names of extreme length. However, some sites go too far by requiring all user names be 7-12 characters in length. Other sites forbid user names that begin with numbers.
A more reasonable approach would be to allow user names from 3 to 16 characters, with a limited set of punctuation allowed, and the first letter cannot be whitespace.
Remember that user names are generally public information so you don’t need to apply the same protections you do to ensure strong passwords. Do the right thing and your users will thank you by not abandoning your account creation form.
Most programmers take a pragmatic approach to security and scale their efforts based on an estimate of the sensitivity of the data they are storing.
The unfortunate truth is that password security is frequently underestimated, making it easy for credentials to be sniffed or stolen. Users often keep a very small collection of passwords, with many people memorizing a small collection and using them on almost every site and service they use. A password compromise on one site can lead to a compromise on many.
The cardinal rule for handling user passwords is to never store the password itself. You should only store a hash (one-way encrypted representation) of the password, cryptographically salted with a string at least 16 characters long. When you want to check a user’s login, simply re-hash their input with the same salt and compare it to the hash stored in your credentials table.
There is never a good reason to display a user’s password on a web page. If they have forgotten their password, send the user through a password reset procedure. This also includes displaying their password in HTML forms. Don’t send the user’s password out as the value of a password form field input. This is almost as bad as displaying it on the page.
On occasion, a site will not use the “password” input field type on a login form. Any sensitive input acting like a password should be presented as a password input. This provides basic protections against caching, copying, and prying eyes. This applies to passwords, PINs, SSNs, and other private authentication data.
Prevent packet sniffing password theft by encrypting pages where the users are changing or creating new passwords.
If you can, SSL encrypt any page where a user is sending their password for login. In places where this may not be an option, consider using javascript to prevent the password from being sent in plaintext. (More info on this in a later blog post)
Many developers don’t consider the fact that cookies are sent by the user’s browser for every request to a domain, including requests for images, CSS, and javascript. All this traffic makes it extremely easy to sniff passwords stored in cookies. Passwords stored in cookies are also easily found by someone who has access to a computer’s harddrive.
All sites need some basic rules around passwords to keep users from using poor passwords. However, don’t think you’re doing yourself or your users any favors if you implement rules like setting a maximum password length, requiring users to change their passwords too often, or preventing users from ever re-using a password once they’ve changed it.
Forcing users into situations like these frequently leads to password post-it notes stuck to monitors or the underside of keyboards. Sometimes, it even drives site users away (I’m talking about you, sharebuilder.com).
Whenever a user’s password changes, send a confirmation message to their verified email address. Tangentially, any email address change should trigger confirmation emails to both their new and old addresses. This behavior helps your users quickly find out when they’ve been compromised, which can limit the damage done by a malicious user.
Many sites feel they are doing their customers a courtesy by emailing them their own password when they change it or sign up. However, if your email account is ever compromised, a simple search for ‘password’ can reveal a treasure trove of passwords, allowing a malicious person to gain access to many more sites and services used by a user. The resultant password list may also be used against any other site that hasn’t emailed a user their password, such as their bank, PayPal, or social networking sites.
When a user forgets their password, generate a random temporary password and email it to their verified email address. This should not overwrite their old password. Instead, it should be set to expire within a reasonable amount of time (a few hours) and if it expires, the old password should remain in place. If the user logs in with the temporary password, they should be required to enter a new password before continuing to the site. The temporary password should then be expired and unusable on that account.
Thanks to the site Feed43.com, I was able to quickly and easily generate an RSS feed to the PHP5 Changelog, a very large page that doesn’t already have a feed.
Check out the PHP 5 Changelog Feed.
Feed43 beats Yahoo’s Tubes service because if a page is too large, it simply truncates it to a usable length. Tubes will simply fail to process a page that it deems is too big.
My Google maps making, RSS feed slinging, universal package tracker has moved to greener pastures. Boxoh.com is your new go-to place for tracking UPS, FedEx, USPS, DHL, and Airborne packages.
Backstory: In 2006, I posted a handy new utility I’d cobbled together which was a mashup between package tracking for for multiple services. It quickly became by far the most popular page on this site, with more than 1.4 million tracking requests last month. It gets more than three times the traffic of my movie theater RSS generator and four times the traffic of another spinoff site, FeedSifter, a simple RSS/Atom feed filter.
If you are familiar with MediaTemple’s GridServer service, you’ll know that using up all 1000 GPUs (server work units) for the past several months is not a good thing. Those cycles weren’t just going to waste on poorly written scripts, either. Each hit to the tracker consumed an average of 0.0002 GPU (WordPress uses 8 to 16 times that with each hit). It wasn’t always this way, though. Check back soon for an upcoming post on how I managed to cut down the CPU usage of the package tracker by 90% with some intelligent code analysis and a creative caching solution.
Boxoh.com is now hosted on a screaming VPS server with plenty of spare power. I’m taking full advantage of APC caching and several other behind-the-scenes tweaks one can only get a grip on when they are running a dedicated server.
Thanks to all of the people who have made the service so popular!
Also, thanks to Juplex for a fast and friendly site design!
As a professional PHP developer who enjoys helping others learn about the language, I have noticed several phases of PHP skill development. One of the first phases is when one stops piling all of their code into one file and starts storing their classes and specialized logic into reusable files. For some, this change brings on a wave of problems that may be very difficult to track down.
At some point, many PHP developers run into an inexplicable White Page of Death or at least stubborn cookies and headers that won’t change when they should. A common cause is include files with white space after the PHP closing tag. Your web server might send these characters to the browser before you process your header() or setcookie() calls, causing them to fail. PHP allows for a single newline after a PHP closing tag before it starts interpreting characters as output.
Omit PHP closing tags
The PHP closing tag is useful when you embed PHP into a page that already contains other content such as an HTML template. However, if you only intend to have PHP logic in a page and nothing else, you can omit the final PHP closing tag:
<?php
$test = new testClass();
$test->helloWorld();
// End of PHP script. No closing tag needed.
This approach does not incur any additional resource penalty to the server.
Output Buffering
If you feel that you need to send output before processing headers and cookies, you should use output buffering. The ob_* class of PHP functions are very useful in these circumstances, but be aware that it comes at the cost of an increased memory footprint and slightly slower processing.
Error Logging
Instead of silently failing or dying with a white page of death, error logging would tell you exactly what is failing and why:
[24-Oct-2008 15:17:31] PHP Warning: Cannot modify header information - headers already sent by (output started at /websites/demo/htdocs/whitespace.php:5) in /websites/demo/htdocs/cookie.php on line 8
I suggest everyone who develops in PHP do so with log_errors enabled. If develop with logging on from the beginning of your project, you will quickly capture potential problems before they grow too great in number and become overwhelming.
I prefer to set my error_reporting level to “E_ALL & E_STRICT” because many standards and best practices are enforced with these messages. (E_STRICT will be included as part of E_ALL in PHP 6)
Once you have logging enabled, you can SSH into your server and “tail” your error log for real-time error output:
[ian@devbox]~ tail -f /tmp/php.log
Using the “-f” flag will follow the log and give continuous output as it appears in the log file.
Please note that logging should be used sparingly (at least at a higher error_reporting level) in production servers. Also, I advise against using PHP’s display_errors in any case, especially production servers. In a production environment, this argument can provide dangerous information to site visitors. You may find it useful in development, but I find that not all errors will be visible or readable when output within your page. However, your log files will never let you down.
I’m working on PCI DSS (Payment Card Industry Data Security Standards) compliance for my company and one of the bigger hurdles we’re looking at is cryptographic key rotation. Our biggest concern is rotating keys for data stored in a DB. It seems we have two solutions and one theoretical option available:
1) 3rd party vendor
There are several companies offering appliances that are essentially crypto proxies which act as a middle man between data and logic.
2) Home brew
We’ve considered storing a key version number next to the encrypted column in the database. When one key is to be disused, a new one is generated and every value stored with the old key revision number is decrypted and re-encrypted with the new key. Meanwhile, all of the data is still accessible as the old key is not invalidated until all rows using that ID have been updated.
3) Theoretical crypto magic
I’m no cryptography expert, but it seems that there should be some means of generating several symmetric keys that result in the same encrypted data. Those keys could then be split into a shared/private pair where the system requesting the data only knows the shared portion; the private portion is a secret known only to the machine performing the encryption. The private key can be invalidated on demand and a new pair generated. No machine need ever store the complete key.
I don’t know if this last scenario is possible. If it’s not out there yet, this may be an interesting market for such a scheme. If there is a workable solution, this may be the ideal solution.
Are there other solutions I’ve overlooked? If you’ve implemented key rotation on DB data, what method did you use?