EPG_Upload

This TAP allows you to upload your own event content into the EPG, as well as timer info, from text files copied to the PVR.

NB This TAP was ported from the SD series TAP created by TonyMy01 - see here for his page. The content on this page still refers in most part to the SD version and how it operates. Over time this will be edited to reflect how it works with the HD version.

Download Link

Use this TAP and the associated PC java applications and uploaders to harvest EPG data from the internet and upload it to your Topfield. Also will upload timers, and now Toppybiff messages. The Topfield-Australia forums [Edit: now defunct] have ideas for how to take advantage of Toppybiff, such as uploading Outlook messages to your Toppy for reading there.

Read the release information notes for more detail on operation of particular features. TED-ICE is the recommended PC software these days for uploading guide and timer information as the Java based grabbers keep getting “broken” by the source guide websites encoding the urls and data on the webpages. You have to subscribe to ICE-TV for TED-ICE to work: http://www.icetv.com.au/.

Installation

Using FTP to transfer the TAP and associated files: Put the TAP preferably into “AutoStart”, the TGDs, and INI settings file into ProgramFiles to begin with, start the tap as normal (hit the filelist button, press the >| or |< button until you find the ProgramFiles folder, hit OK to go into the AutoStart folder, find the EPG Uploader X.XX TAP, press OK). Putting a TAP into AutoStart will auto launch the TAP on Topfield power up. If you have any problems during ANY TAP loading (misconfiguration etc) that causes the Topfield to hang after startup, you should press the “0” key repeatedly on startup (until you see the screen aspect change is good enough) which will stop all TAPS loading. You can then launch FTP and move or delete the offending TAP out of the “AutoStart” folder, or use the Topfield’s menus to move it.

The TAP will load and save the TAP epg_upload.ini file to the ProgramFiles folder (however if you have it in auto start it will detect this and delete the ini file and save it in the ProgramFiles folder automatically).

To bring up the TAP menu, hit exit key with nothing on screen, and next tap/cancel option on each tap you have running until you find the menu for the EPG Upload TAP (if the TAP is busy processing files, you won’t be able to bring up a TAP menu). TMSCommander can also be used to bring up the menu, available from my website (UHF key will bring up the TSRCommander menu). Choose the menu option to load the EPG files up, either one at a time or all files. Auto Start is definitely the best directory to put it in, as then every time the Topfield boots up, it will load up the daily tgd files from the “TGD directory=EPGdata” directory, making this a fairly automated process if you set the EPG Uploader menu to “AutoLoad EPG: YES”. To bring up the TAP menu options again, press the EXIT key without any menus on screen (or the UHF key if you are running TSR_commander). This will bring up most TAP menus, most are set to propagate this EXIT along to the next TAP if you have multiple TAPs running by choosing the default “CANCEL”/”Next TAP” option in the other TAPs.

This TAP supports TGD files in a folder directly off the main root drive, the specific ini file entry is called “TGD directory=” and will need editing on your PC before uploading. If it is not set in the ini file, it will default to the “ProgramFiles” directory, the provided inifile is setup for “EPGdata” which matches the default TED/TEDS PC uploader inifile settings. Put the directory name in the ini file (no “\” or “/” slashes, case sensitive), this should keep your ProgramFiles folder clean if you choose to use this option (the log file and TSV will also save to this folder). The manually specified directory/folder MUST be off the main root folder of your drive, unless you want to use “ProgramFiles” or “Auto Start” in the inifile (the TAP interprets those folder names correctly, you don’t need to specify with “/” or “\”).

GMT offset should be set to your local GMT offset (+11hours eastern daylight, +10 eastern standard, etc). If the EPG data you are feeding the Toppy is GMT (rather than local time), then set the offset to 0 (the current JavaXMLTV feed takes info from local time websites, so the GMT time needs to be set). Newer versions of this TAP now support using the GMT_offset.ini file to determine the offset, this is created by ProgressBarKeys(PBK) TAP, to automate the offset setting during such things as daylight savings.

Edit the INI file with relevant information for your region. The parameters in it are for Sydney, using Logical Channel Numbers LCN 2,7,9,10,3 for the channels to have EPG info displayed for (new ini file has the video program guide channels included too). The “service_name” field is a text string field that MUST MATCH (including the case!) the tv channel name in your harvested TGD files, and is case sensitive. It is NOT the LCN name for your channel, or any name that appears in the Topfield channel lists. This name depends on what area you get the JavaXMLTV to harvest from. I advise harvesting your first weeks worth of programs, and then look in the XML or TGD files for the name of the channel the programs are on in your area .
Apart from the channel details and “TGD directory=” information, all other information in the ini file SHOULD be setup in the TAP menu.
If a Toppybiff message comes up (email icon and tone indication) hit the OK key to read it. If you keep pressing the OK key, it will page through the message. If you hit the EXIT key, it will close the viewer and this will also EXIT will clear the message indication without reading. Images of Toppybiff in operation are here:

The Toppybiff wav files must be named toppybifflow.wav and toppybiff.wav and be in the same directory as the logfile, TGD files etc.
Use the included TAP option “Channel service details” to discover your own service number details (original network id, transport stream id, service id). This is becoming less important as TED takes all the hard work out of setting this stuff up.

If you aren’t running TED, but are running a variant of JavaXMLTV to harvest, you must also edit the PC JavaXMLTV “javaxmltv.prop” file to get the correct program guide information for your area. The file has information on what number to put in for the area you live in. You are best to edit these INI and prop files with something like WordPad. Notepad doesn’t do a good job for unix files like the .prop file, the formatting will not look right with Notepad.

Menu Options

Menu Option Explanation
Use GMT_Offset.ini Yes/No option to use the ProgressBarKeys TAP (PBK) generated offset file, which will auto adjust for daylight savings
GMT offset the offset between the EPG time and GMT, for Australia East Coast it is normally +10hours and +11 in daylight savings time (the inifile entry is number of 15min intervals if you wondered what the 40/44 in the demo file meant!). Shows DST or not if the above GMT_Offset.ini option is set, and won’t allow to be altered if this is the case.
Now/Next update the offset between when the program is due to start, and when you want the now and next information to start showing the next program. -1 is the default option (meaning from 0 to 1 minutes before the EPG entry is due to start, the now/next will tick over to the next program). This is good for, say, if you setup a timer to start 3mins before a program is due to start, if you set the update to -4mins, the timer will pick up the new program details. If you cycle to +31 it will read “NO” which is used to stop the TAP from trying to update the Now and Next status (suitable for, say, ICE tap users or people running broadcaster N&N updates)
New file check freq TS off sets the frequency the tap should check for new TGD files (it bases these intervals on time division, so 00:05 will be 5mins past the hour, 10past, 15 etc). New files are only loaded if a manual load has been triggered, or AutoLoad is turned on, unless they are 9999XXXX.tgd toppybiff or timer details. This will read the Topfield timeshifting parameter and if the TS is off, will use this setting.
New file check freq TS on As above, but if the TS is ON, will use this parameter. The reason for these two options is that filechecking spins up the HDD if it is spundown due to having timeshift off, so can be annoying to regularly have the HDD spin up. PBK has an option to turn on and off the timeshift in peak and offpeak viewing times, so this parameter will let you set 2 different frequencies to attempt to load new timers, toppbiff and EPG more regularly with timeshift on.
Email Volume Options are LOW, HIGH and OFF. For the Toppybiff email/message notification.
Load individual EPG files Brings up a list of your tgd files, so you can load them one at a time.
Load ALL EPG files Will load all tgd files.
Build epg_timers.tsv file Is used to manually create a timerlist file that TED/TEDS can use to avoid timer conflict. This file is created everytime a TGD file is loaded, but this menu option lets you manually create it. Version 2.3 onwards also creates this file whenever epg_uploader sees the number of timers has changed.
AutoLoad EPG will load all tgd files when the TAP is launched (including from auto start on bootup), and regularly check for new EPG files at the New File Check Freq. If this option is set to NO, the TAP will still regularly check for Toppybiff and timer TGD files, this setting is suitable for those who want the timer functionality of the TAP without all the EPG functionality, such as ICE tap users.
Delete Existing EPG at Startup Used to delete the Now&Next or whatever is loaded prior to the TAP loading, note, normally in Australia this should be set to yes, unless you run another uploader like ICE and want to mainly use the ICE tap.
Reload Hour For those who keep their Toppies on 24x7 and constantly update the EPG information downloaded, this option will let you choose an hour that the TAP will delete all the existing EIT and reload the TGD files. Choose 25:00 if you don’t want to use this feature.
Load Days Loads only a number of TGD files, rather than all of them, to speed up loading and only show the EPG details for the number of days specified.
Show Loading Info stops the onscreen loading and timer creation messages. The messages are still logged into the logfile. You will see onscreen messages with individual file load still. Options NO, MINIMAL and FULL. MINIMAL will give an indication of loading events in the top left corner, and show loaded timers with “Cxx Fxx Dxx” format, where C=created, F=failures, D=deleted
Logging Level 4=Log everything, including my debug stuff I put in the code. 3=Show most information including all onscreen messages logged, 2=show some, 1=show minimum necessary to see loaded events and timers, 0=not recommended (no logging at all). Reason included- keep unnecessary stuff out logfile for normal use, also reduced disk usage (suitable for people running no timeshift for example and want the HDD to spin down). Recommended level for timeshiftON=3, timeshiftOFF=2. 4 is suitable for reporting problems or seeing the most information.
Broadcaster EIT BLOCK/LOAD Block is default, if you want to allow the broadcaster event information (transmitted by the broadcaster) loaded, you can set this to “LOAD” (remember if it has Now & Next info this will overwrite your uploaded EPG entries confusing things a little with overlaps).
EPG load wait 0s- set the load wait time for each EPG file from 0 to 10secs, suitable to either get the loading done as quickly as possible or to slow it down so you can change channels/volume/etc between file loads (not so important now with systemProc regularly called and thus the remote can be pressed during most epg_uploader operations).
Load ICE INI NONE: normal tap behaviour. NO: similar to NONE I guess, except the tap will assume a 2nd ini exists and will add an entry to the inifile. YES: TAP will load “epg_uploadice.ini” file rather than “epg_upload.ini“.
TEDS Recordings File Will append to “epg_recordings.tsv” for each new recording started if set to YES.
Reload TAP Will reload the tap similarly to when starting the TAP. Useful when swapping the inifile or settings.
Channel service details brings up the list of service, original network and transport stream IDs for all your channels, press the menu key to return back to the tap main menu, or exit to close the service details window completely.
IniFile Interpreted Viewer shows the values that the TAP believed it read from the inifile, suitable for debugging inifile problems (remember to have carriage returns between your entries, and a carriage return after the last entry in your inifile). Everytime you bring up the main TAP menu, it will reload the inifile, so if you make changes to the inifile, you don’t need to quit the tap. Press up/down to scroll, exit to close and menu to bring up the main TAP menu again
LogFile Viewer will show the full 10kbyte logfile (wrapped to fit). Defaults to showing the last entry, you can scroll up/down between entries. Press exit to close and menu to bring up the main TAP menu again.
Exit quits tap
Next TAP brings up any further tap menus for other taps. You can now use TMSCommander with this tap to save propagating through a number of tap menu’s to get to this menu.

The TAP supports a method to enable you to load up a TGD file called “99999999.tgd” which will allow you to put well formed entries in it (same format as normal TGD files) suitable for setting timers. This could be done by remotely accessing your PC, edit/build 99999999.tgd and let your PC schedule or Topfield FTPD server or whatever load the file up the same way as the other TGDs are loaded. TED/TEDS takes advantage of this to set timers (details for TED/TEDs below).

The TAP logs everything to a “epg_upload.log” file, including successful (or failed!) timer creations, so now you can remotely create 99999999.tgd files or simply look back on any errors. This file is created in the same place you have the TGD files stored. There is also a TAP “LogFile Viewer” that will read the log. The log is truncated at 10240bytes, but the TAP will always add new events to the end of the file and erase bytes from the beginning to keep the filesize the same. The logfile viewer defaults to displaying the last event, you can scroll up through all the events.

The TAP supports having 2 of the same service IDs (for instance, Nine Sydney and Win Wollongong). Previously (prior to 2.5) the tap used the service_id only to tie in timers and N&N updates to the epg entries, it now uses service_id, transport_stream_id & original_network_id to determine the correct Topfield channel, so from version 2.5 no longer needs the YYY part of “service_name=XXX,YYY”. This improves loading speed as well…. LCN (logical channel number, a DVB-T standard extension) support is also provided, negating the need to have the serviceIDs in the inifile at all. A typical entry for LCN support is:

[Channel 1]
service_name=ABC
LCN_channel=2

The TAP saves a snapshot of the timers in a TSV (filename “epg_timers.tsv”) everytime it uploads any TGD or other file (TSV tab seperated variables can be imported into a spreadsheet package like Excel very easily), and also saves this file each time it detects the number of timers has changed.

The TAP supports loading the same TGD/EPG record to multiple channels (such as propogating to the VPG video program guide channel). However, timers added by 99999999.tgd file will only be to the FIRST service_name= service in the list.

Simply add the service_id information for your other channels, like:

[Channel 2]
service_name= Channel Nine Sydney
LCN_channel=9

[Channel 3]
service_name= Channel Nine Sydney
LCN_channel=99

(notice the service_name, this is the TGD channel name detail, is the same for both services).

The satellite options since version 1.27 requires the satellite name in the inifile to correctly support loading to the satellite channels.
An example of this is here:

[Channel 2]

service_name=C2.sfdrs.ch
satellite_name=Hotbird
service_id=907
transport_stream_id=8500
original_network_id=318

[Channel 3]

service_name=dsf.com
satellite_name=Astra
service_id=900
transport_stream_id=33
original_network_id=133

So the TAP will reload the TGD files for up to two satellites if you change channels to the new satellite (unfortunately the API seems to mainly support the Terrestrial Topfields, but if you change channels to a particular satellite, the uploading will work fine for that satelllite, thus the TAP will load once for the satellite you are tuned to when first running the TAP, then when you change channels to a different satellite, it will reload the TGD files).

The TAP has two inifile support, where the TAP will bootup with epg_upload.ini and redirect to load the epg_uploadice.ini file if setup in the TAP menu to “Load ICE INI: YES”. This feature was introduced to allow simple comparison between EPG source suppliers for example (especially if you make sure the two “TGD directory=“ entries are in different directories). If you have multiple LCNs for the one broadcaster (e.g. LCN1 and LCN10 for channel 10), you can put one in the epg_upload.ini and one in epg_uploadice.ini and setup a favourites entry to allow easy moving between the 2 channels in your EPG view to do an instant comparison. It may also be useful for people who move between two regions regularly.

The TAP supports a maximum of 30 services.

Download Version http://tonyspage.abock.de/TAPS/JavaXMLTVetc.zip to get the original harvester apps (with a Nov2005 mod to accommodate source encryption at the time… although this keeps changing these days, keep an eye on the Topfield-Australia forums for which grabbers people are using at the time), although now there is a new harvest/uploader (was still using PeterU’s JavaXMLTV a while ago, seems to be using other harvesters now) called TED http://teds.trappett.com/ which includes options to build html guide files on your PC from the info, automatically uploads, sets timers etc. Also, TED supports downloading ICE EPG now for those subscribed to ICETV here in Australia. There are “season pass” mods to both TED which is called Topfield EPG Driver’s Seat (TEDS) from the same site (and now included with the TED download), and mods to the original uploader “tfSeasonPass”, these let you search on guide keywords and setup timers based on them. TEDS is getting heaps of development, and I now recommend that one, as it will do things like combine adjacent programs, add start and end delays etc etc. I highly recommend TED/TEDS to basically record all your favourite programs based on keywords. I haven’t had to setup a manual timer in a very long time since running TED.
If you are going for the non-TED solution mentioned above (JavaXMLTV), you will need the Java runtime engine (JRE) running for the JavaXMLTV harvester to work. This is downloadable from: http://java.sun.com/j2se/index.jsp (choose the Download J2SE JRE link unless you are wanting to do java development, in which case you can download Download J2SE SDK).

To download EPG information from the net, if you are using the original and/or modded JavaXMLTV files (provided by myself for the TAP, JohnS for the exe file/VBS uploader, PeterU for the JavaXMLTV) I created a “DOEVERYTHING.bat” file that will do the works. You must not have Altair running for this to work (I use an application called KillProcess to kill Altair as part of the DOEVERYTHING file, although TED makes this redundant now, as it will kill and launch Altair for you).

Read the other readmes and look at what all the batch files achieve to understand more about the harvesting aspects of this solution (TED or the original uploader).

Warning: The current Topfield API (the environment a TAP uses) is a little buggy. Things like, when it deletes timers, after a reboot one may show back up again, and issues like, when it creates timers, after a reboot, all but one of them may be forgotten! I have been given some indication that approx 3.5 seconds has to be kept between any timer creating or deleting events (from the GUI and from the API), so later versions of this TAP maintain this delay between timer create/delete events, and I have not ever seen this issue crop up since maintaining this gap.

TGD (Topfield Guide Data) file format

Multiple carriage return separated records, with entries tab separated:

service_name^tYYYY/MM/DD HH:MM^tduration mins^ttitle^tshort event desc^textended event desc^tparental^trecording status^teventidCR

Where ^t is a tab
Where parental is M/G/PG/R/C/MA
And recording status is:D=delete, Y=yes, +/-XX +/-YY = extend XX start/YY end minutes, ED is to record every day, WD for weekday, WE for weekend, WL for weekly and anything else won't record
Eventid is optional, will one day be used to erase or alter particular entries.
service_name matches the name obtained from your XML feed.

Fault Finding

!!!!!!!!!!!!!!! Number one thing to check for any problem !!!!!!!!!!!!!!!!!!!!!!
Make sure that you look at the “IniFile Interpreted Viewer” first, and double check that you have the correct number of services, and no silly information like “zeros” for any of the ID entries (probably caused by either no CR/LF “the enter key” after EVERY entry line, especially check the last entry has one, or the “Number of Services” number doesn’t match the actual entries you have placed in the file). !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
No guides are loading - First check you have the correct folder specified in both the TED ini and the Topfield INI file (TGD directory=XXXX) for where it should locate the TGD files. Do a “Load individual EPG files” option from the TAP menu to verify it can see the TGD files, and use Altair to check where the files are located.
Files show they are loading, but no info appears in the guide - This could be that you have chosen the wrong area to get the guides from for TED/JavaXMLTV, or put the wrong information into the epg_upload.ini file. Make sure the TAP has got the same information that you think you have put into it, by observing the “IniFile Interpreted Viewer”. Carriage returns need to be at the end of every line in the inifile, with gaps between some entries, so this could be a reason for the loading problems, it will look clear when you look at the viewer (or you could copy the file back to your PC and look at it there, the TAP re-writes the INIfile each time the TAP menu is closed). Set the Logging Level to 4 and restart the TAP, observe what messages are sent to the logfile. Log file messages like “2005/05/09 19:13:36-More than one Topfield service_id:1234 found” and “2005/05/09 19:13:36-Can’t find inifile service_id:0 for any Topfield channels” indicate definite inifile entry issues.
Now&Next or Timers are being created for the wrong channel - The TAP now scans all the Topfield internal channels on startup to match them to the information in the inifile. If you rescan or add or delete Topfield channels while the TAP is running, and you end up with no more or less channels, the TAP won’t rescan for the new information (it does detect deleted or added total number, and will rescan the channels). Restart the TAP in this case.
Now&Next is not updating - There is 2 firmware releases for most variants of Topfield around the November and December 2005 period that will not allow the API to read the Topfield EPG information (it shows all corrupted information for any TAP that reads it, such as EPG TAPS that read the EIT, or say, improbox that reads the EPG to show the N&N. EPG_uploader also reads the EPG table to determine whether to trigger the Now&Next updating, so you cannot run this era firmware and have EPG_uploader behave correctly for N&N. Another possible issue is that your clock may jump back greater than 3 shows period of time in the EPG, and if this is the case, it may not update correctly too, so alter the parameter in the infile to “Scan ALL Events=YES”.
My Toppy is crashing on bootup, help!!! - Don’t panic, during bootup simply press the “0” remote key repeatedly which will stop the TAPS from loading. You can then sort out the reason for crashing (two main reasons, corrupt ini file or corrupt TGD information), and restart the TAPS or Topfield to restart them. You should not get any entries in the log file like “2005/05/09 13:03:57-Can’t find inifile service_id:0 for any Topfield channels”, this suggests you have a problem with an inifile channel entry, and when a timer or epg entry is created for this channel, this could cause a crash.
The timers are loading for the wrong time- did you make sure the GMT offset was setup correctly? The inifile is number of 15min intervals, the menu setting is obvious. I really recommmend setting all settings up from the menu (of course the TGD directory and channel details must be hand entered into the inifile). This offset is applied to the XML/TGD data to convert it to “GMT” which is the required format for EIT records. It is also used for the Now&Next update. So if your source and local region are in 2 timezones, a future version of this tap will hopefully sort this out!

Technical Info

Maximum number of services supported: 30

Maximum TGD record size: 5000bytes

Maximum TGD field size: 3984 bytes (although obviously some fields are strict format, like the recording info etc). Maximum extended information text size is 3735 bytes due to the EIT standard, anything more will be truncated at EIT load time. 3984 is the Toppbiff max for the main email body/item, most fields are truncated if they exceed the EIT standard now, which is typically 256bytes for short event description and name etc.

Expected filenames:

up to 8_numeric_chars.tgd for tgd files (numeric mainly to help sorting), in specified directory in the ini file
99999999.tgd file handled a little differently, it doesn’t become part of the “which is the latest file to be uploaded” algorithm, and can simply be used to set timers (so with the Load 99999999 event info=NO option will simply load up the timer information without feeding the event details to the EPG). In “TGD directory=” folder.’
99999997.tgd file used for “Toppybiff” as discussed here: http://www.nslu2-linux.org/wiki/Puppy/ToppyBiff
YYYYMMDD.tgd are the actual EPG files that contain the harvested EPG data, and must, like the other tgd format files, be located in the “TGD directory=” folder.
epg_upload.ini for the ini file, in the “ProgramFiles” folder.
epg_upload.tsv for the timer file created whenever an upload is done, in “TGD directory=” folder.
epg_upload.log created/appended whenever a message is sent (also to the serial port if you want to look there). The logging level dictates the amount of messages that get to it. In “TGD directory=” folder. Truncated to 10240bytes, new information always appended (oldest information is truncated first).
epg_recplay.tsv TAB separated variable file defined in the release notes Version3.9 below in the “TGD directory=” folder, showing playback or recording information.
epg_recordings.tsv TAB separated variable file that accumulates the recording information by appending the REC1/2 info to the file as shown above for epg_recplay.tsv.
epg_dtimer.tsv is a file used to indicate there are deleted timers, stored in the ProgramFiles folder, currently its presence is detected by the timer_extend8.5 and newer version tap so that if you have “Check future timers: ON” set for Timer_Extend this will trigger timer extend to reset the one time timer backup (deletes any backups newer than the current date) so that the tap doesn’t attempt to restore any timers deleted by EPG_uploader.
epg_uploadice.ini for a second ini file, in the “ProgramFiles” folder, that can contain a second set of parameters for comparison purposes etc.
toppybiff.wav audio file used to indicate toppybiff messages have arrived, stored in “TGD directory=” folder. Format advised to be 16bit stereo 48kHz sampling rate.
toppybifflow.wav as above, but suggested to be a lower volume or less obtrusive audio file, used when the menu option for “Email Volume” is set to LOW.

Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License