Reviewing group policy options for a user

I’m continually changing group policy options in my organization so its useful to actually test these options rather than logging on each time with a test account. Recently I found a way to do this very easily!

Instructions

Start Menu >> mmc

File >> Add/Removed snap in

Pick the “Resultant set of Policy”and double click to add it.

Click “Resultant set of Policy” on the tree.

On the toolbar click >> Action >> Generate RSoP Data….

I like to use planning mode as this is live on your AD infrastructure.

Choose the AD container your user is in, then the AD container the computer is in

Then click Next for all the other screens.

Voila! You can now see any GP settings applied to any user or computer without needing to log them on!

Improving Web Service API’s

I recently popped into the IRC channel of a well known online music service to flag up a few errors that have been happening to their API web service. I am a big user of this service and appreciate it immensely, but simply wasn’t able to get a reliable connection to it using various apps.

I’m a big believer in improving things gradually over time and must say over the last 10 years I’ve learned a lot about running an efficient web service. While speaking with the music service guys, I suddenly became the user rather than the developer (which was kind of nice to see things from the other side!) so thought a general blog post about what I’ve learned may be useful to this site, and indeed others.

Having run sites like TheAudioDB, TheLogoDB, TheSportsDB and Movie-XML in the past I have a fair bit of experience running these large services for millions of users. We handle a good 15 million requests on peak days like Sunday from the various metadata sites, so I’ve had to improve our services to keep up with demand. The failure of Movie-XML made me appreciate how important it is to scale a website with demand quickly. The site became a huge success overnight and basically failed due to not being able to serve API requests quickly enough. These days TheMovieDB does a far better job than I did, and does it well.

These days we only use about 10% of our server resources at peak times, and hopefully this should be able to scale to 100’s of millions of API hits a day in the future before we need to look again. With TheSportsDB gaining popularity recently with the live scores features, this scaleability will be important for the future.

So here are my top tips for creating a fast web server API:

Hardware

Separate the API from the front-end website on a physical level. This can mean using a dedicated server, or leveraging a cloud service, but its important to decouple the demand for your website with the demand for your web service. Your website may be doing much more complicated operations that can cause database waits that can have a knock on effect to the API.

Invest in good web server hardware. I run my own co-located equipment and as a server admin by-day, try to use the latest hardware technology. Our web server uses an Intel PCIE SSD based storage which is blazingly fast and a Xeon 6 core CPU. IO bottlenecks of traditional mechanical storage can have a huge knock on effect on CPU and server resources.

Match your development hardware to your server hardware. While you can’t always get a complete match, make sure it uses similar technology and software versions. I found that using different versions of PHP for example, caused performance problems when transferred onto the main web server.

Software

The best piece of advice I can give to anyone running a popular API is to look at the use cases of your users. Typically you will have a few, very large API users who are doing the same thing over and over on your service to fulfill a simple need. Analyze this data meticulously and improve the web-service accordingly.

Check the search functionality of your API. Analyze the metrics for how long each search takes and how accurate it needs to be. Databases offer all kinds of different search algorithms but does the web service need such accuracy? Does it need 50 results instead of 5? How does it handle duplicates? There are lots of things you can do to simplify the data returned to the point where it is far more efficient.

Database design is hugely important to speed and accuracy of a web service API. Look at the amount of tables, improve indexes and links between tables, and look at the amount of data that needs to be scanned. Full table lookup’s are bad, too many joins are bad, using certain sort types like filesort are bad. If you really want to take it to the extreme you can create a table specifically for the API output and simply do a single lookup of cached results. I’ve found this to be a useful tool to speed up database reads.

Transferring to JSON as a web service helped a lot as its quicker and smaller than XML, while still being universally known to developers. We depreciated our XML API about 5 years ago and at a guess it improved bandwidth and resource usage by a good few percent.

When it comes to MYSQL there are loads of different options for speeding up the database. I spent countless hours tweaking these to use the correct level of caching queries, memory usage, table types and concurrent connections. The MYSQLTuner script is useful for this but don’t just trust it, experiment! Its only by trial and error you can get this right, it depends on how your database is structured and used.

Managing usage

Lets face it, analyzing millions of log entries in a text file is no fun! Use a live SQL analyzer to monitor your API and peak times. This way you can see the slow queries in real time and adapt your code to make it more efficient. In my case, I removed a few advanced features(such as alternative artist name searches) and it made a huge improvement. You can always allow this kind of search with a special URL, just not by default. I personally use Jet Profiler for MySQL which is a great visual way of seeing which sites, tables and queries cost the most resources. Jet Profiler gives you a list of the top 10 slow queries as well as charts showing usage over time and even gives advice on how to improve the SQL. Its a great tool well worth the money but there are many other free and open source alternatives.

I know all to well as an amateur developer its easy to write something, but much harder to get into the minds of other developers. API’s tend to be static systems that the 3rd party developer has to adapt to. This is a good thing, but its worth going back to those developers and asking what they would like to see. Over the years I have added many new API methods to my sites after feedback from the user forums. Some of these new methods work far better for the 3rd party developer than previous attempts and in some cases have really improved performance as well as features.

API keys

Analyze usage constantly. Who’s using your API? What times are they using it? Why are they using it? Are any users causing problems or significant loads? Revoke API keys if necessary, or use this as a tool to force changes. Each piece of software using your API should have a specific key, larger users should change their keys every couple of years to prevent piggy backing from unauthorized apps. Its also important to give out keys for different software process even if its the same overall user. This helps track down problems when they show up and pinpoint them quickly.

If your writing data back to your API, make sure its efficient. Writing and creating indexes can take a lot of CPU in some cases. Use API tokens (essentially temporary user API keys) to achieve tracking like this and limit the user base to only those who really need it.

Upgrade the API – Over time it may become apparent that the original API doesn’t cut it and needs to be replaced. Its a great idea to version your API, so v1 for the initial release, then v2 with some extended features, v3 for performance improvements ect. Over time developers will migrate to the new API and you can also run them in parallel for a transitional period.

Documentation

Although probably unpopular with most developers, writing good documentation is key to getting a reliable API service. This is because many developers may make mistakes or take shortcuts when initially writing their software. If you have good documentation guiding them then it helps to reduce these mistakes and let them get the data they need with the minimum API calls. I’ve even gone as far as looking at the 3rd party developers code on places like github to suggest better ways of using our API’s.

Documentation isn’t just about showing your API syntax, try showing some real world guides or examples of other applications using it. Make a sandbox so the developer can quickly test the API with dummy data. I came accross a sports data api recently that limited api lookups to once an hour for some methods. Maddening for the developer just starting out!

Artwork

We serve a lot of artwork over our API and it became apparent that a huge amount of bandwidth was being consumed this way. We recently moved all our artwork to a sub domain, and linked it to the cloudflare so it could be cached using a content delivery system. This saves us a couple of terabytes a month in data fees! Best of all its currently free!

Hopefully the above helps if you are building a new API or improving a current one.

Making Gimp look like Photoshop

As a few of my regular readers know, I hate a bright interface so I was looking around how to make GIMP look darker. I found a theme online that looks rather like Photoshop and is much easier on my eyes while doing logo work for www.theaudiodb.com and www.thesportsdb.com

As you can see, I also use the single window interface as its much easier to work with on the desktop.

To install

1. Download the theme from here
2. Copy the “PrudenceDark” folder to C:\Users\Username\.gimp-2.8\themes
3. Relaunch GIMP

I don’t really understand why GIMP don’t provide a dark theme as default but this is a great way to make it look great, and keep the familiar icons.

PrettyPrint for PHP

As I get better at coding PHP, I’m starting to realize the real power of being able to debug things quickly on the fly. I’m also a great believer in making things look good at the same time.

So I was looking for a way to output an XML file in a debug window like other languages in a nice way. In the past I had always just var_dump, or echo’d the variable but this can end up pretty ugly to say the least on long input streams.

So I found out about print_r and the <pre> tag and made a small function.

// Function to show an array nicely
function prettyPrint($pretty) {
    print "<pre>";
    print_r($pretty);
    print "</pre>";
return $pretty;
}

All you need to do now is something like this in your main code page

$xml = simplexml_load_file('test.xml');
prettyPrint($xml);

Thats it! A nice prettyPrint function for PHP 🙂

Source: http://stackoverflow.com/questions/5393085/display-an-array-in-a-readable-hierarchical-format

Renewing an SSL certificate

I recently had to renew one of our 3 year old SSL certificates we use at work for a website. I thought it would be a straight forward invoice from the certification company and they would renew. How wrong could I be…

Once we purchased our renewal, the SSL text was sent via email but it seemed to be a completely new certificate. For some reason when I tried to install the intermediary certificate, then complete the private CRT import inside IIS 7, I kept getting an error:

ASN1 bad tag value met. 0x8009310b

Ahh Microsoft error messages at their best! Basically this error means it cant verify the intermediary certificate against the new private SSL we just purchased. After lots more digging, it basically turned out the private SSL I had double clicked and installed had gone into the Current User/Personal certificates location, rather than the Local Computer/Personal certificates location.

Once I manually moved the certificate into the Local Computer location inside the certificate MMC, it all worked perfectly running the repair command from the cmd prompt. This also automatically imports it into IIS 7 on the main website.

certutil -repairstore my “THUMBPRINT_OF_CERTIFICATE”

Source: https://support.comodo.com/index.php?/Default/Knowledgebase/Article/View/656/17/asn1-bad-tag-value-met-0x8009310b

 

Office 365 Password Expiry

Recently I’ve been getting a lot of users coming to me with forgotten passwords, I thought this was strange because most of them have been using the same passwords for years. I found out that they were actually all asked to change passwords last year and a fair amount had forgotten the details since. This is annoying as I believe passwords should be secure over time, It feels a lot more insecure when people are forced to change them and keep forgetting them all the time.

So I went looking for a way to set passwords to never expire on Exchange online. Here’s how to do it:

Download: http://www.microsoft.com/en-us/download/details.aspx?id=41950
Download: http://go.microsoft.com/fwlink/p/?linkid=236297

Launch “Windows Powershell” from the start menu

To connect to Exchange Online type and enter your email admin details:

Connect-MsolService

And then you can check an individual users expiry setting using this command:

Get-MSOLUser -UserPrincipalName test@test.com | Select PasswordNeverExpires

And set individual user accounts to never expire with this command

Set-MsolUser -UserPrincipalName <user ID> -PasswordNeverExpires $true

Or set the entire organization with this command

Get-MSOLUser | Set-MsolUser -PasswordNeverExpires $true

So there you have it, an easy way to change the password expiration policy on Office 365!

Review of the QNap 453s-pro SSD NAS

I’ve used Qnap devices for a few years now and been really happy with the 419p+ box that I have. Its got a great user interface and “just works” with 4 x 2tb hard disks in it the capacity is about 6tb.

Since I’m a huge advocate of SSD’s and the larger drives have reached a good price level now, I decided to buy a smaller qnap NAS to fill with 1tb Samsung SSDs (see the smaller box on the right in the title picture). This makes a total storage size of about 3tb with RAID enabled.

The setup was incredibly easy, I just popped the drives in the device and booted it up. You need to use the Qnap finder software to identify the temporary IP address, then give it a static IP and name. Once that’s done, I upgraded the firmware to the latest version and checked out some of the Qnap plugins available. I setup sabnzbd (which is a newsbin downloader), couchpotato, sickbeard and headphones. Since the NAS runs 24/7 and has no spinning disks any more, this software works wonderfully now.

I checked the power usage compared to my old 419p+ mechanical drive NAS and it uses around 30% of power. I reckon that’s going to save me a fair bit of money over the years with this device! The biggest advantage I find is that the device is now totally silent, the spinning up of disks used to drive me crazy. Not only that, but the NAS is tiny! It is so small you could put it anywhere, its also safe to move around, and the SSDs are so reliable, I now have complete confidence in them. Not to mention the speed! I’m getting about 30mb/s over wireless copying files which is a great improvement over the old NAS of about 10mb/s.

Overall this is one of the best purchases I’ve ever made! I’ve now turned the other 2 NAS’s off, and just use this one as a daily media storage.

Adding a character to the end of a field using MYSQL

I was recently looking for a quick way to add a comma to the end of a MYSQL field. My first thought was to loop it through a PHP script and update the string for each field. This seemed like a long way round and I always love a good 1 line SQL statement fix so here it is.

update table set field = concat(field,’character to update’);

Setting up Notepad++ to look like CodeAcademy

I love a good looking IDE and recently came across a nice website called codeacademy.com. They have a beautiful colour scheme, so I tried to replicate it on my own notepad++ setup. Here’s how…

NotePad++ >> Settings >> Style Configurator >> Style >> Default Style

Now change the default style font (not the global override) to “Courier New”, and to size 12 (or size 14 if you use a 1080p screen). You should now see a nicely spaced font with quite big text.

Lastly we need to apply a new theme to Notepad++. The theme is called “tomorrow_night” and can be downloaded from here:

https://github.com/chriskempson/tomorrow-theme

Simply copy the notepad++ specific XML files into the file explorer url location:

%APPDATA%

/notepad++/themes

NOTE: This location will depend on your set-up, sometimes it can be in the program files/notepad++ location if you chose that option on install.

Since the theme is available for lots of editors, you can also do this for SublimeText 3.

Happy (beautiful) coding

Flipping “surname, firstname” to “fullname” with just MYSQL

I love a good quick data fix and recently found this one on stackoverflow.

Say you have a MYSQL field with this format:

Surname, Forename

and you want to get into a new field:

Forename Surname

You can do this from the SQL prompt:

# Add new column
alter table $TABLE add $newField varchar(40); # or something like that

# Populate new data
update $TABLE set $newField = CONCAT(
  SUBSTRING_INDEX(SUBSTRING_INDEX($oldField,',',2),',',-1),
  ' ',
  SUBSTRING_INDEX($oldField,',',1)
);

Great stuff and very quick too!

Source – http://stackoverflow.com/questions/2654716/how-do-i-flip-a-column-in-mysql-that-has-data-structured-last-name-first-name