Feature #3190

Implementing over-the-air EPG grabber for stateside ATSC broadcast (PSIP)

Added by G E almost 2 years ago. Updated almost 2 years ago.

Status:FixedStart date:2015-10-21
Priority:NormalDue date:
Assignee:Adam Sutton% Done:

0%

Category:EPG
Target version:4.2

Description

Hello,
I just finished with a fellow from the #htc IRC channel, username lsd he was remote into my system here and put together a file from the local channel Muxes for parsing info to hopefully implement EIT for hts.
EIT would be a great addition to TVHeadend here in the states and would be much easier to setup then using XML grabbers, setting up scripts and cron jobs to keep it updated. I have used Myth and it sees the local channel EIT information; but Myth is, in my opinion, too much to have to go through to set up to use with Kodi or even it's own frontend.
I hope the attached file helps. If not contact me and I will put together another file.
Thank you
gee

History

#1 Updated by Jaroslav Kysela almost 2 years ago

  • File deleted (test.ts)

#3 Updated by G E almost 2 years ago

#4 Updated by luuthe . almost 2 years ago

This might be a good place to start, but I've yet to experiment with it.

https://github.com/laurimyllari/tvheadend/tree/4.0-atsc-epg

#5 Updated by B C almost 2 years ago

luuthe . wrote:

This might be a good place to start, but I've yet to experiment with it.

https://github.com/laurimyllari/tvheadend/tree/4.0-atsc-epg

oh great, so no need to actually reinvent the wheel. @perex: take a look at the psip module

#6 Updated by G E almost 2 years ago

One more for good measure. This is from a different local channel.
https://www.dropbox.com/s/4ct61li5jg05463/wfla-test.ts?dl=0

#7 Updated by Jaroslav Kysela almost 2 years ago

I sent an e-mail to Lauri to sign CLA and create the pull request so we can adapt his code.

#8 Updated by Lauri Mylläri almost 2 years ago

Jaroslav Kysela wrote:

I sent an e-mail to Lauri to sign CLA and create the pull request so we can adapt his code.

After replying to you I found this issue, so I'm commenting here directly.

I created a pull request at https://github.com/tvheadend/tvheadend/pull/734

Can you see from this if I already signed the CLA - I think I did when I started working on this.

#9 Updated by G E almost 2 years ago

This question will show my lack of knowledge in this.
"_dvb_support: change GPS UTC offset to one less …_"
Is this syncing the time up and if so could you use a NTP server for time sync or the broadcast station's timecode which should be in the data stream.
Just curious how the time syncing is done.

#10 Updated by G E almost 2 years ago

Also, maybe look at myth's code. The EIT information parsed and then displayed within Kodi seems to be on time and accurate.
Sorry if I am being a pain. Just throwing ideas up in the air here.

#11 Updated by B C almost 2 years ago

you can let tvh sync your time from the networks if you have a computer without internet, but you must not.

#12 Updated by B C almost 2 years ago

G E wrote:

Also, maybe look at myth's code. The EIT information parsed and then displayed within Kodi seems to be on time and accurate.
Sorry if I am being a pain. Just throwing ideas up in the air here.

G E, it seems Lauri has all you need, just be a little bit patient for getting his PR merged and then we see if anything needs to be fixed. Don't worry about accuracy, that's the smallest problem.

#14 Updated by G E almost 2 years ago

Oh I'm in no rush here just inquisitive on some of this and also just pulled some references that I used when I used to work in broadcast.

#15 Updated by Jaroslav Kysela almost 2 years ago

  • Subject changed from Mux Information for implementing EIT for stateside ATSC broadcast EPG to Implementing over-the-air EPG for stateside ATSC broadcast (PSIP)
  • Target version set to 4.2

#16 Updated by Jaroslav Kysela almost 2 years ago

  • Subject changed from Implementing over-the-air EPG for stateside ATSC broadcast (PSIP) to Implementing over-the-air EPG grabber for stateside ATSC broadcast (PSIP)

#18 Updated by Jaroslav Kysela almost 2 years ago

I merged with some fixes the code from Lauri, so we need testers now :-) v4.1-724-gc08bf56

#19 Updated by luuthe . almost 2 years ago

Tested latest master (rev-list#7104).
The EPG is populated successfully, times are correct along with show names and descriptions.
Great work Lauri and perexg!

#20 Updated by G E almost 2 years ago

Thank you very much Gentleman on a job well done.
This is the fastest that I have ever seen a group of Devs go to work on a request for a upgrade or fix. Everything is looking great so far.

I have just one small suggestion; and this is probably due to my lack understanding of the program.
In the EPG panel view, second column over, "Progress", the title can be a bit confusing, well at least to me. I thought at first it was the progress of the download of EIT information and I discovered that is the percentage of the program time past.
My suggestion, the title of the second column be changed to Time, Time Past or change the count to show the time that is left and call it Time Left. If this can be done also replace the "%" with "m" with the math to show actual minutes.
Just a thought.

Again thanks for getting the OTA/EIT working.

#21 Updated by Chris Lee almost 2 years ago

Here is 7 ATSC recordings from all 7 channels in Edmonton Alberta Canada.

https://drive.google.com/open?id=0B7PjDRC3SaJxZEh1SVRZZkVqZDg

I'll install tvheaded and test what I can as well.

UDL

#22 Updated by Mark Clarkstone almost 2 years ago

Chris Lee wrote:

Here is 7 ATSC recordings from all 7 channels in Edmonton Alberta Canada.

https://drive.google.com/open?id=0B7PjDRC3SaJxZEh1SVRZZkVqZDg

I'll install tvheaded and test what I can as well.

UDL

Tried two of those TS files and they both failed to play in VLC. It reports..

core debug: processing request item: atsc_2.ts, node: Playlist, skip: 0
core debug: resyncing on atsc_2.ts
core debug: atsc_2.ts is at 0
core debug: starting playback of the new playlist item
core debug: resyncing on atsc_2.ts
core debug: atsc_2.ts is at 0
core debug: creating new input thread
core debug: Creating an input for 'atsc_2.ts'
core debug: requesting art for atsc_2.ts
core debug: using timeshift granularity of 50 MiB, in path 'C:\Users\Mark\AppData\Local\Temp'
core debug: `file:///D:/Downloads/atsc_2.ts' gives access `file' demux `' path `/D:/Downloads/atsc_2.ts'
core debug: specified demux `any'
core debug: creating demux: access='file' demux='any' location='/D:/Downloads/atsc_2.ts' file='D:\Downloads\atsc_2.ts'
core debug: looking for access_demux module matching "file": 12 candidates
core debug: looking for meta fetcher module matching "any": 1 candidates
core debug: no access_demux modules matched
core debug: creating access 'file' location='/D:/Downloads/atsc_2.ts', path='D:\Downloads\atsc_2.ts'
core debug: looking for access module matching "file": 21 candidates
filesystem debug: opening file `D:\Downloads\atsc_2.ts'
lua debug: Trying Lua scripts in C:\Users\Mark\AppData\Roaming\vlc\lua\meta\fetcher
core debug: using access module "filesystem" 
core debug: Using stream method for AStream*
core debug: starting pre-buffering
lua debug: Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\fetcher
core debug: received first data after 0 ms
core debug: pre-buffering done 1024 bytes in 0s - 1000000 KiB/s
core debug: looking for stream_filter module matching "any": 6 candidates
core debug: no stream_filter modules matched
core debug: looking for stream_filter module matching "record": 6 candidates
core debug: using stream_filter module "record" 
core debug: creating demux: access='file' demux='any' location='/D:/Downloads/atsc_2.ts' file='D:\Downloads\atsc_2.ts'
core debug: looking for demux module matching "any": 66 candidates
lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\fetcher\tvrage.luac
lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\fetcher\tvrage.luac
core debug: no meta fetcher modules matched
core debug: searching art for atsc_2.ts
core debug: looking for art finder module matching "any": 2 candidates
lua debug: Trying Lua scripts in C:\Users\Mark\AppData\Roaming\vlc\lua\meta\art
lua debug: Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\art
lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\00_musicbrainz.luac
lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\00_musicbrainz.luac
lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\01_googleimage.luac
lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\01_googleimage.luac
lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\02_frenchtv.luac
lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\02_frenchtv.luac
lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\art\03_lastfm.luac
lua debug: skipping script (unmatched scope) C:\Program Files\VideoLAN\VLC\lua\meta\art\03_lastfm.luac
core debug: no art finder modules matched
core debug: art not found for atsc_2.ts
qt4 debug: IM: Setting an input
ts debug: eof ?
ts debug: Force Seek Per Percent: Seeking failed at 10%.
ts debug: Force Seek Per Percent: PCR's not found,
ts debug: pid[49] unknown
ts debug: pid[65] unknown
ts debug: pid[48] unknown
ts debug: pid[68] unknown
ts debug: pid[7424] unknown
ts debug: pid[7680] unknown
ts debug: pid[8187] unknown
ts debug: PATCallBack called
ts debug: new PAT ts_id=16429 version=9 current_next=1
ts debug: * number=1 pid=48
ts debug: PMTCallBack called
ts debug: new PMT program number=1 version=9 pid_pcr=49
ts debug: * es pid=49 type=2 fcc=mpgv
core debug: selecting program id=1
ts debug: * es pid=52 type=129 dr->i_tag=0xa
ts debug: * es pid=52 type=129 dr->i_tag=0x81
ts debug: found language: eng
ts debug: * es pid=52 type=129 fcc=a52 
ts debug: * es pid=53 type=129 dr->i_tag=0x81
ts debug: * es pid=53 type=129 dr->i_tag=0xa
ts debug: found language: enm
ts debug: * es pid=53 type=129 fcc=a52 
core debug: using demux module "ts" 
core debug: looking for a subtitle file in D:\Downloads\
core debug: looking for decoder module matching "any": 43 candidates
avcodec debug: CPU flags: 0x0107d3db
avcodec debug: trying to use direct rendering
avcodec debug: allowing 4 thread(s) for decoding
avcodec debug: avcodec codec (MPEG-1/2 Video) started
core debug: using decoder module "avcodec" 
core debug: looking for packetizer module matching "any": 23 candidates
core debug: using packetizer module "packetizer_mpegvideo" 
core debug: looking for decoder module matching "any": 43 candidates
core debug: using decoder module "a52" 
ts debug: DEMUX_SET_GROUP 0 0000000000000000
core debug: looking for meta reader module matching "any": 2 candidates
lua debug: Trying Lua scripts in C:\Users\Mark\AppData\Roaming\vlc\lua\meta\reader
lua debug: Trying Lua scripts in C:\Program Files\VideoLAN\VLC\lua\meta\reader
lua debug: Trying Lua playlist script C:\Program Files\VideoLAN\VLC\lua\meta\reader\filename.luac
core debug: no meta reader modules matched
core debug: `file:///D:/Downloads/atsc_2.ts' successfully opened
ts debug: pid[7425] unknown
ts debug: pid[7427] unknown
ts debug: pid[7682] unknown
ts debug: pid[7683] unknown
ts debug: pid[7681] unknown
ts debug: pid[7426] unknown
ts debug: eof ?
core debug: EOF reached
core debug: finished input
core debug: removing module "avcodec" 
avcodec debug: ffmpeg codec (MPEG-1/2 Video) stopped
core debug: killing decoder fourcc `mpgv', 0 PES in FIFO
core debug: removing module "packetizer_mpegvideo" 
core debug: removing module "a52" 
core debug: killing decoder fourcc `a52 ', 0 PES in FIFO
core debug: removing module "ts" 
ts debug: pid list:
ts debug: - pid[0] seen
ts debug: - pid[48] seen
ts debug: - pid[49] seen
core debug: Program doesn't contain anymore ES
ts debug: - pid[65] seen
ts debug: - pid[68] seen
ts debug: - pid[7424] seen
ts debug: - pid[7425] seen
ts debug: - pid[7426] seen
ts debug: - pid[7427] seen
ts debug: - pid[7680] seen
ts debug: - pid[7681] seen
ts debug: - pid[7682] seen
ts debug: - pid[7683] seen
ts debug: - pid[8187] seen
ts debug: - pid[8191] seen
core debug: removing module "record" 
core debug: removing module "filesystem" 
core debug: dead input
core debug: changing item without a request (current 0/1)
core debug: nothing to play
qt4 debug: IM: Deleting the input

#23 Updated by Ron L almost 2 years ago

I only looked at atsc_1.ts but I think perhaps only the psip PID was recorded, not the raw mux with video and audio.

#24 Updated by G E almost 2 years ago

I'm not sure if this is an issue with EPG/EIT or due to our time change and the local stations.
We just switched our time here in the US one hr back DST (Daylight Savings Time). The system clock here is on the right time.
The EPG is apparently broken again according to the browser interface. It is only getting some of the stations information as it did earlier on as the EIT was being worked on. It was working up to a few days ago Wednesday,but this evening is the first time since the time change that I had a chance to check on it. I also just took an update for TVheadend earlier today.
v. 4.1-812~ge85a3c7
I did restart hts also tried clearing the information so hts can repopulate the information.

#25 Updated by Jaroslav Kysela almost 2 years ago

This issue was fixed in v4.1-822-g18d993f . Update.

#26 Updated by G E almost 2 years ago

I just took a upgrade from htc v4.1-822-g18d993f to v4.1-830~gaeb20ca
Still no go.
Damn DST.

#27 Updated by G E almost 2 years ago

Scratch last... It took a few minutes over an hour to populate all the channels.
Kodi EPG is now populated.
Thanks again

#28 Updated by Jaroslav Kysela almost 2 years ago

  • Status changed from New to Fixed

#29 Updated by Jaroslav Kysela almost 2 years ago

Just a little note: I fixed many things in v4.1-894-ga27b0af regarding the psip parser - events should be fetched much faster and the description text should be fetched, too (it didn't work for many events). v4.1-894-ga27b0af

Report issues to bug #3272 .

#30 Updated by Jaroslav Kysela almost 2 years ago

Just a little implemntation note: Only 5 PIDs are used to scan ATSC EIT/ETT tables now (the subscribed EIT/ETT tables are changed in a loop). It might fix some issues with tuners which are not capable to do hardware filtering of many PIDs.

Also available in: Atom PDF