4

I am looking for getting all games played in 2013 as quickly and painlessly as possible. A batch file to download many ziped pgn files would be non-ideal, but ok. I prefer the Week in Chess as a source due to their high DB consistency quality, but the ultimate goal is to compile a single high quality DB of all games played in the year 2013.

  • Besides a TWIC download? If you're willing to wait + pay, the megabases that Chessbase releases in 2014 will pretty much cover TWIC for 2013 (as well as every other year before that! :)) – shivsky Dec 1 '13 at 3:32
5

Here is your requested zip: 2013_TWIC.zip

I created a command script for this. See attached. I assumed you are running Windows, but the script is easily adapted to other OSes.

For downloading, it uses curl. You can get that at http://curl.haxx.se/download.html, and for Windows, I recommend the msi installer.

For unzipping, I used Java's Jar tool, which is free. But use whatever you prefer.

When running the script, run dos shell as Administrator in Win8 or Win7. At the end of the script it puts all PGNs into one large PGN file called 2013_TWIC.pgn HTH.

Click: Script to get 2013 TWIC PGN files

3

An easy solution in C++:

    #include <urlmon.h>

    #pragma comment(lib, "urlmon.lib")

    const int firstnumber = 920, lastnumber = 1104;
    const char * const Spath = "http://www.theweekinchess.com/zips/twic"; // source path
    const char * const Dpath = "C:\\Users\\<username>\\Downloads\\twic\\"; // destination path

    void main()
    {
        int x;

        for (x = firstnumber; x <= lastnumber; ++x)
            URLDownloadToFile(nullptr, (Spath + std::to_string(x) \
    + "g.zip").c_str(), (Dpath + std::to_string(x) + ".zip").c_str(), 0, nullptr);


    }

A powershell unzip script:

$shell = new-object -com shell.application

for ($x = 920; $x -le 1103; $x+=1) {

    $zip = $shell.NameSpace(“C:\Users\<username>\Downloads\twic\$x.zip”)

    foreach($item in $zip.items())
    {
        $shell.Namespace(“C:\Users\<username>\Downloads\twic”).copyhere($item)
    }

}

A simple command line pgn file merge: copy *.pgn twic.pgn

3

I was trying to solve the same problem but since I am not familiar with C++ I did it in Python instead. My program also unzips and extracts the PGN files from the downloaded zip files.

Code:

  import pycurl
  from io import BytesIO
  import os
  import zipfile
  from pathlib import Path

  c = pycurl.Curl()

  for i in range(start, end + 1):

    url = "http://theweekinchess.com/zips/twic" + str(i) + "g.zip"

    save_location_path = Path("/home/paluchasz/Desktop/chessbase_stuff/downloading_twic/twic" + str(i) + ".zip")
    unzipped_location_path = Path("/home/paluchasz/Desktop/chessbase_stuff/downloading_twic")

    with open(save_location_path, 'wb') as f:

        c.setopt(c.URL, url)
        c.setopt(c.WRITEDATA, f)
        c.perform()

    with zipfile.ZipFile(save_location_path, 'r') as myzip:
        myzip.extractall(unzipped_location_path)

    os.remove(save_location_path)

  c.close()

Notes:

  • Firstly to find the url I just put my mouse pointer on the download pgn button on the TWIC website. I noticed that the urls have a common structure and the only thing that changes are the numbers. However, that url appeared to be incorrect as I was getting a 301 (moved permanently) error. As it turned out it moved to the same url but without the 'www' at the start.
  • Adjust the range function depending on what files you want to download, so if you want to download from 1100 to 1250 then set start = 1100 and end = 1250
  • The unzipped_location_path should be the path to the folder where you want the pgn files to be saved to. The save_location_path will be a path to where the zip files will be downloaded to first (I chose the same folder for this). These also need a file name at the end of the path which needs to be different for each zip. Hence the str(i) + ".zip" ending.
  • It is not necessary to use Path() - it's use is that this script can now be run on another operating system which perhaps has the same path (Windows for instance uses backslashes instead and this path would not work)
  • The pycurl library downloads the files. Zipfile then reads these files and extracts them. Finally os.remove(save_location_path) removes the no longer useful zip files.
  • Finally, I originally downloaded CBV Chessbase files instead. However, I could not see an easy way to combine all those files together and I would have had to append to my database one file at a time. That is why I downloaded PGN instead as now I could use the cat command ($ cat * >twic1100-1250.pgn) to merge all the files into one.

Automating the process

  • Thanks for sharing! Does this script work on Windows or Linux or both? Paths look like Linux, but you mention ChessBase:) – Joe Nov 2 '18 at 3:02
  • I think it should work on both although I only tried on Linux! The point of Path() is so it works on both but you could just use the windows path with backslashes instead and then it should work. I am using Linux but I have a Windows in Virtual Box - that's why I mentioned ChessBase. I have a shared folder between the two systems and that's where I save the pgn files! – Szymon Palucha Nov 3 '18 at 13:35
1

Write a script that downloads the files, one at the time. The filename structure will allow you to use a loop to capture all the files you wish to get. Then, write a script that unzips the files and concatenates them in a single large PGN file. Then, write a script that finds and discards duplicated games. Other strategies seem to include a Firefox add-on (https://addons.mozilla.org/en-US/firefox/addon/downthemall/) and in general, the topic has been discussed in other places (http://www.chess.com/forum/view/general/13-million-master-games), (http://www.chess.com/forum/view/chess-equipment/quotlost-issuesquot-of-twic), (http://chessdb.sourceforge.net/tutorial/twic-retriveal.php).

protected by Phonon Nov 1 '18 at 17:52

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.