Pi in the Sky

DDRUK.com stepfiles collision data

Posted on July 2nd, 2007 at 10:36am by Pi.
Categories: Development, Gaming, Untranslated.

This is collision data of the stepfiles for DDR simulators (Stepmania, Dance With Intensity and similars) found in DDRUK.com. This data allows for detecting equal steps in seemingly different stepfiles.

DDRUK.com has a huge, very complete and very exact Dance Dance Revolution stepfile collection for DDR simulators, in DWI format. DDR players are often tempted into downloading everything and playing everything. PA players (Perfect Attackers) often want to get through every song, get full combos, etc.

However, some of these songs are repeated. Detecting duplicate songs (the audio) is rather easy; but it’s not always easy to determine whether the steps for two duplicated songs are the same, or if they’re different and worth playing.

I’ve prepared this collision data to show where the steps are the same. When steps are the same, I call this a “collision”. All this data has been made using all the stepfiles from all the bumper packs available at DDRUK.com dated on or before April 24th, 2007 (2066 stepfiles). I used to do this manually with a simple text editor, and a spreadsheet program (specifically, Crimson Editor and MS Excel). All it took was a bit of batch file and Excel formula fiddling. Now I’ve created a set of PHP scripts which do all this in a snap.

Where the step data begins with empty beats (strings of zeroes), the empty beats have been deleted. There are several files which have the same steps except that one has extra empty beats at the beginning (this is caused by same songs using different mp3 which have different amounts of silence at the beginning). Otherwise, the step data is untouched. Although I only use single steps, I’ve included double, couple and solo steps because it’s trivial.

While doing all this stuff, I’ve spotted a few errors in the stepfiles. For example, “Electrical Parade” from Disney Mix for GB has DISPLAYTPM instead of DISPLAYBPM. Several songs are exactly the same, except that their mp3 have different qualities (notably the Festival songs from Ultramix). Tittle Tattle from Ultramix 2 has two Double steps in the same line. ORION.78 (AMeuro-MIX) 01 from DDR Edit Data 5th Mix has also two steps in the same line, although one of them is commented. PARANOiA KCET (clean mix) from DDR 2nd Link Version has one steps (couple smaniac) with invalid (?) step characters, specifically “5″, which is not a valid step character according to the DWI documentation. Two songs have triple steps in single mode (three arrows at the same time: Uh La La La (single maniac) from Dancing Stage 1.5, and Skulk (single smaniac) from Dance Dance Revolution Ultramix 2. There’s an extra semicolon in one of the couple steps in Only You from 4th mix.

The contents of the collision data archive are mainly four tab-delimited text files. These files are easily importable into any database or spreadsheet program. Each one of these files is the output of one of my scripts, so I’ll talk about both the files and the scripts at the same time.

The first script is ‘dwicollisions.php’, which reads all the .dwi files in the current folder (or in an specified folder through a command-line argument) and processes the data inside them. First of all, it creates the file dwidata.txt, which contains all the needed data from the DWI files; not all the data, just the data needed to do the collision detection and processing.

Each row of this file corresponds to one step set of a DWI song. It has some data about the steps; the steps themselves, mode and difficulty, the name of song from where they came, etc. Additionally, the last column indicates if the steps have a collision; if the content is equal to 1, then there’s another row with the same steps.

The script then generates a second file, called dwicollisions.txt. This file contains again the step data, but only of the songs which have at least one row colliding with another row. There are more extra columns, which specify in detail with which rows or songs each step or song is colliding.

Note that while in dwidata.txt the steps are untouched, while the steps present in dwicollision.txt are unpadded. The collision detection for dwidata.txt was done with unpadded steps too, but I thought it could be useful to have somewhere the original, untouched stepdata.

The columns of these files are pretty self explanatory. Each different DWI gets assigned an unique ID, and each steps (row) another unique ID.

Some of the columns in dwicollisions.txt are a bit more complex, here they are explained:

• groupid: each unique set of steps which are duplicated gets an arbitrary ID assigned.
• grouprowcount: how many rows with the same steps are in the group.
• grouprows: comma-separated list of all the rows in this group, including the current one.
• groupsongcount: how many songs with the same steps of this group there are.
• groupsongs: comma-separated lists of all the songs which contain the same steps in this group, including the current one.
• colwithsongcount: the number of other songs that steps of this song collide with, including current song.
• colwithsongs: at song-level, comma-separated list of all songs with matching collisions.

The second script, ‘dwicollisionsanalysis.php’, uses the data in dwicollisions.txt (which must be in the same folder when you run the script). At this moment, all it does is just simple analysis. First, it compares simple 1 to 1 collisions: songs with one or more steps which collide with steps in only another song, and only if the other song only collides with the first. It generates the file dwicollisionsanalysis1to1.txt, with quite self explanatory columns too. There’s a column indicating which steps collide, another with which steps are only in the first song, and another one with which steps are only in the second song. To make it easy, each step has been reduced do a single identificator in the form of mode-difficulty-feet, like single-maniac-7.

Sometimes, the same steps would have different mode-difficulty-feet identifiers. For example, the single challenge steps of one song could be exactly the same as the single heavy steps of another song, or simply have different feet ratings - all this while both sets of steps are exactly the same. In this case, and for the column indicating which steps collide, it’s used the identificator of the first song.

For the rest of the songs which aren’t one to one collisions (non-circular, or more than two song collisions in the same songs), there’s some very simple analysis. The script creates dwicollisionanalysisother.txt and lists collisions from a single song point of view. It tells, for each song, which steps are repeated elsewhere, and which steps are unique in that song. Songs with steps which only repeat in themselves aren’t considered “with collisions”. I found 11 of these songs; for example, the double standard and the double maniac steps are the same.

For more detail about the scripts, open them in any text editor and read the beginning comments.

Now you can use all this data to decide wether a song is worth keep and play, or if it’s not. In my case, I’m only interested in single heavy or challenge steps - so if two songs do not have new, different heavy or challenge steps in single mode, I delete one of them. That way I know I’m not replaying steps I’ve already done, and I know that all I have is worth playing. More or less.

However, there are other uses. For stepfile creators, it’s a good testing tool. Some steps are supposed to be duplicated across different stepfiles. These scripts, or all this data, can easily tell where and how collisions happen, and stepfile creators can be sure that they replicated the steps in the right places without painfully loading pairs of dozens of DWI files.

As an extra, there’s another PHP script called ‘dwilist.php’, which I wrote to generate a list of songs in a particular way. It has nothing to do with collisions, but I thought it could be good to include it. To run these scripts, you need PHP 4.3 or higher (I used 4.4.6 to develop them) installed. The scripts can be run from the command line and some accept parameters - read the notes at the beginning of each script. These scripts come without any warranty. And the data comes without any warranty too. Total disclaimer, no liabilities, no fit to any purpose, etc. Public domain and all that. I’d like to know if you got to use any of these, beep me if you actually use the data or scripts.

Well, I think that’s pretty much all. There’s actually a lot more about it, but I’ll cut it here. I hope this data can be of help for players, collectors, and stepfile creators. If so, please leave a comment! I might update this data whenever DDRUK.com updates their bumper packs, but don’t count on it. Feel free to distribute it, or spread the word linking to this article.

OK, I’ve reached the end. Here’s the actual zip file (2.0mb):
DWI Collisions version 1 (2007-07-02) in zip format.

no comments yet.

A Scanner Darkly »« Sometimes, one receives nice things in emails

Pi in the Sky is powered by WordPress. Dressed with Vistered Little. Hosted at MochaHost.