Feature #3182

Custom user agent for automatic iptv (m3u)

Added by Njuskalo Njusko about 2 years ago. Updated almost 2 years ago.

Status:FixedStart date:2015-10-20
Priority:NormalDue date:
Assignee:-% Done:

100%

Category:IPTV
Target version:4.2

Description

Sometimes, a custom user agent is needed for playing iptv streams. When using pipe/ffmpeg, a command line such as pipe://ffmpeg -user-agent Kodi/14.1 ... was sufficient.

It would be nice if network or mux menu had a selectable or custom user agent string available.

trace.log (8.97 KB) Njuskalo Njusko, 2015-10-29 20:42

trace1.log (9.25 KB) Njuskalo Njusko, 2015-10-29 20:58

trace4.log (10.4 KB) Njuskalo Njusko, 2015-10-30 09:22

trace5.log (9.47 KB) Njuskalo Njusko, 2015-10-30 15:26

trace6.log (9.3 KB) Njuskalo Njusko, 2015-10-30 21:24

trace7.log (9.47 KB) Njuskalo Njusko, 2015-10-30 22:04

trace8.log (9.47 KB) Njuskalo Njusko, 2015-10-30 23:10

trace9.log (9.47 KB) Njuskalo Njusko, 2015-10-30 23:15

Associated revisions

Revision be7381f1
Added by Jaroslav Kysela almost 2 years ago

IPTV: allow custom HTTP headers for muxes, fixes #3182

History

#1 Updated by Jaroslav Kysela almost 2 years ago

  • Target version set to 4.2

#2 Updated by Jaroslav Kysela almost 2 years ago

  • Status changed from New to Fixed
  • % Done changed from 0 to 100

#3 Updated by Jaroslav Kysela almost 2 years ago

Fixed in v4.1-789-gbe7381f . Syntax for 'Custom HTTP header' is:

User-Agent: my user agent

Note that any HTTP header can be added (one per line - multiple lines are parsed).

#4 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

Fixed in v4.1-789-gbe7381f . Syntax for 'Custom HTTP header' is:

User-Agent: my user agent

Note that any HTTP header can be added (one per line - multiple lines are parsed).

Perfect, many thanks. Will try and report.

#5 Updated by Njuskalo Njusko almost 2 years ago

Njuskalo Njusko wrote:

Jaroslav Kysela wrote:

Fixed in v4.1-789-gbe7381f . Syntax for 'Custom HTTP header' is:

User-Agent: my user agent

Note that any HTTP header can be added (one per line - multiple lines are parsed).

Perfect, many thanks. Will try and report.

Not working.

#6 Updated by Njuskalo Njusko almost 2 years ago

Tried with stream that blocks Kodi, User-Agent: Kodi/14.1, but the stream plays normally

#7 Updated by Njuskalo Njusko almost 2 years ago

Njuskalo Njusko wrote:

Tried with stream that blocks Kodi, User-Agent: Kodi/14.1, but the stream plays normally

The above was tested with pipe://

Now I tried with simple http:// link and User-Agent: Mozilla/5.0 but it failed as well.

#8 Updated by Jaroslav Kysela almost 2 years ago

Try v4.1-794-g3e1e39c . The problem was that the default User-Agent was not removed, so the request contained two User-Agent headers.

#9 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

Try v4.1-794-g3e1e39c . The problem was that the default User-Agent was not removed, so the request contained two User-Agent headers.

Still doesn't work.

#10 Updated by Jaroslav Kysela almost 2 years ago

Show me '--trace httpc' .

#11 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

Show me '--trace httpc' .

Please find attached

#12 Updated by Jaroslav Kysela almost 2 years ago

2015-10-29 20:14:48.843 [  TRACE]:httpc: 47 45 54 20 2F 74 73 2F 48 52 54 32 2F 73 74 72 GET /ts/HRT2/str
2015-10-29 20:14:48.843 [  TRACE]:httpc: 65 61 6D 5F 77 69 66 69 5F 31 32 30 30 6B 2E 6D eam_wifi_1200k.m
2015-10-29 20:14:48.843 [  TRACE]:httpc: 33 75 38 20 48 54 54 50 2F 31 2E 31 0D 0A 48 6F 3u8 HTTP/1.1..Ho
2015-10-29 20:14:48.843 [  TRACE]:httpc: 73 74 3A 20 63 64 6E 2E 69 70 74 76 2E 61 6D 69 st: cdn.iptv.ami
2015-10-29 20:14:48.843 [  TRACE]:httpc: 73 2E 6E 65 74 0D 0A 43 6F 6E 6E 65 63 74 69 6F s.net..Connectio
2015-10-29 20:14:48.843 [  TRACE]:httpc: 6E 3A 20 63 6C 6F 73 65 0D 0A 55 73 65 72 2D 41 n: close..User-A
2015-10-29 20:14:48.843 [  TRACE]:httpc: 67 65 6E 74 3A 20 4D 6F 7A 69 6C 6C 61 2F 35 2E gent: Mozilla/5.
2015-10-29 20:14:48.843 [  TRACE]:httpc: 30 0D 0A 0D 0A                                  0....           

GET /ts/HRT2/stream_wifi_1200k.m3u8 HTTP/1.1
Host: cdn.iptv.amis.net
Connection: close
User-Agent: Mozilla/5.0

Seems ok ? But the m3u file is not parsed. Show me the contents of it..

#13 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

[...]

Seems ok ? But the m3u file is not parsed. Show me the contents of it..

#EXTM3U
#EXT-X-VERSION:4
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:1349183
#EXTINF:10,
/ts/HRT2/2015-10-29/1210211/HRT2_wifi_1200k-1349183.ts
#EXTINF:10,
/ts/HRT2/2015-10-29/1210211/HRT2_wifi_1200k-1349184.ts
#EXTINF:10,
/ts/HRT2/2015-10-29/1210211/HRT2_wifi_1200k-1349185.ts
#EXTINF:10,
/ts/HRT2/2015-10-29/1210211/HRT2_wifi_1200k-1349186.ts
#EXTINF:10,
/ts/HRT2/2015-10-29/1210211/HRT2_wifi_1200k-1349187.ts

#14 Updated by Jaroslav Kysela almost 2 years ago

Try v4.1-796-g2e92cca . I added application/x-mpegurl as valid mime type for m3u.

#15 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

Try v4.1-796-g2e92cca . I added application/x-mpegurl as valid mime type for m3u.

Thanks. 4.1-796~g2e92cca produces an error. Please see the log.

#16 Updated by Njuskalo Njusko almost 2 years ago

Njuskalo Njusko wrote:

Jaroslav Kysela wrote:

Try v4.1-796-g2e92cca . I added application/x-mpegurl as valid mime type for m3u.

Thanks. 4.1-796~g2e92cca produces an error. Please see the log.

Configured without --enable-libffmpeg_static and it works now.

#17 Updated by Njuskalo Njusko almost 2 years ago

Posted in wrong issue thread. Please disregard the above.

#18 Updated by Njuskalo Njusko almost 2 years ago

Njuskalo Njusko wrote:

Njuskalo Njusko wrote:

Jaroslav Kysela wrote:

Try v4.1-796-g2e92cca . I added application/x-mpegurl as valid mime type for m3u.

Thanks. 4.1-796~g2e92cca produces an error. Please see the log.

Configured without --enable-libffmpeg_static and it works now.

Still doesn't work.

#19 Updated by Jaroslav Kysela almost 2 years ago

Could you try v4.1-798-g8185796 ?

#20 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

Could you try v4.1-798-g8185796 ?

Crashed. Please see the attached --trace httpc log.

#21 Updated by Jaroslav Kysela almost 2 years ago

Try v4.1-803-g48f04e3 .

#22 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

Try v4.1-803-g48f04e3 .

Still not working. Please see the log.

#23 Updated by Jaroslav Kysela almost 2 years ago

v4.1-805-g08e57f0

#24 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

v4.1-805-g08e57f0

4.1-808~g2cab577

Still not working.

#25 Updated by Jaroslav Kysela almost 2 years ago

v4.1-809-gcb72dbf

#26 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

v4.1-809-gcb72dbf

Not working.

#27 Updated by Jaroslav Kysela almost 2 years ago

trace7 is from 808

#28 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

trace7 is from 808

How so ?

2015-10-30 22:02:44.007 [ NOTICE]:START: HTS Tvheadend version 4.1-809~gcb72dbf started, running as PID:2780 UID:1000 GID:1000, CWD:/home/tommy/tvheadend CNF:/home/tommy/.hts/tvheadend

#29 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

trace7 is from 808

4.1-811~g4900abd

Not working as well.

#30 Updated by Jaroslav Kysela almost 2 years ago

I'm sorry - working on many things together. I need break. Try v4.1-812-ge85a3c7 .

#31 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

I'm sorry - working on many things together. I need break. Try v4.1-812-ge85a3c7 .

Didn't work with v4.1-812-ge85a3c7

#32 Updated by Jaroslav Kysela almost 2 years ago

v4.1-820-g496a1b9

#33 Updated by Njuskalo Njusko almost 2 years ago

Jaroslav Kysela wrote:

v4.1-820-g496a1b9

The stream can be played now, great work, thanks.

The syntax for custom HTTP headers supports escape chars, i.e. 'Kodi/14.1' ?

#34 Updated by Mario D almost 2 years ago

Is it supposed to work a channels like the following with both the headers in auto IPTV net?

#EXTINF:-1 tvg-id="0003" group-title="Deutsch" tvg-logo="0004.png",[COLOR orangered]ZDF HD[/COLOR]
http://wownet.ro/str/0004.m3u8|X-Forwarded-For=81.17.18.50&User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0

Do I need escaping? It doesn't stream for me.

#35 Updated by Jaroslav Kysela almost 2 years ago

@Mario: Try v4.1-826-g902f307 (if it does not work - show what TVH has in the custom HTTP headers field in mux).
@Njuskalo: TVH parses whole strings until the delimiter is found (space or | ). No escaping. I'm following these examples: http://xmtvplayer.com/build-m3u-file .

#36 Updated by Mario D almost 2 years ago

@Perex: it continues to fail for me on scan.
The content of the 'custom HTTP headers' in mux is like:

X-Forwarded-For=81.17.18.50&User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0

It is the supposed format?

It could help to test a similar m3u list on your system: http://pastebin.com/V4YiW8Zv
The listed channels all worked in a test few minutes ago on kodi using simple IPTV add-on.

#37 Updated by Jaroslav Kysela almost 2 years ago

@Mario: Try v4.1-827-g55aa465 . The lines should be (without &):

X-Forwarded-For=81.17.18.50
User-Agent=Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0

#38 Updated by Mario D almost 2 years ago

Ok, now the headers in the field have the expected format.
The mux scan still end with a "FAILED" status, but the service and the channel are created. They work as expected: is it normal that the mux has "FAILED" status?
All the channels in the pastebin list work.

#39 Updated by Jaroslav Kysela almost 2 years ago

I need to see debug logs - there should be lines with 'complete' and 'incomplete' for DVB service information tables at the end of the scan procedure. Something like:

2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 1D4C (7500) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 1E14 (7700) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 1DB0 (7600) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 1BBC (7100) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 1B58 (7000) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 012C (300) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 00C8 (200) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 19C8 (6600) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 0064 (100) pmt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 0011 (17) sdt complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 0010 (16) nit complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 0001 (1) cat complete
2015-11-01 17:58:09.324 [  DEBUG]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network - 0000 (0) pat complete
2015-11-01 17:58:09.324 [   INFO]:mpegts: Multiplex [onid:233A tsid:4085] in TSfile Network scan complete

#40 Updated by Mario D almost 2 years ago

Forcing scan on the provided ZDF mux:

nov 01 18:12:30 sauron tvheadend[3756]: mpegts: test iptv - adding mux prova.m3u - ZDF in test iptv to scan queue weight 6 flags 4000
nov 01 18:12:30 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - add raw service
nov 01 18:12:30 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - tuning on IPTV
nov 01 18:12:30 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - started
nov 01 18:12:30 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID 0012 (18) [20/0x122da100]
nov 01 18:12:30 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID tables subscription [0042/0x122e0ba0]
nov 01 18:12:30 sauron tvheadend[3756]: subscription: 0134: "scan" subscribing to mux "prova.m3u - ZDF", weight: 6, adapter: "IPTV", network: "test iptv", service: "Raw PID Subscription" 
nov 01 18:12:33 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID 0000 (0) [20/0x7f0168003ad0]
nov 01 18:12:33 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID 0001 (1) [16/0x7f0168004f40]
nov 01 18:12:33 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID 0010 (16) [16/0x7f016800bd50]
nov 01 18:12:33 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID 0011 (17) [20/0x7f016800d1c0]
nov 01 18:12:33 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID 0011 (17) [16/0x7f016800e630]
nov 01 18:12:33 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - open PID 0FFF (4095) [16/0x7f01500013f0]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID 0FFF (4095) [16/0x7f01500013f0]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID 0011 (17) [20/0x7f016800d1c0]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID 0010 (16) [16/0x7f016800bd50]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID 0001 (1) [16/0x7f0168004f40]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - 0000 (0) pat incomplete
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - scan no data, failed
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID tables subscription [0042/0x122e0ba0]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - stopping mux
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID 0000 (0) [20/0x7f0168003ad0]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID 0011 (17) [16/0x7f016800e630]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: prova.m3u - ZDF in test iptv - close PID 0012 (18) [20/0x122da100]
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: test iptv - removing mux prova.m3u - ZDF in test iptv from scan queue
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: test iptv - adding mux prova.m3u - ZDF in test iptv to scan queue weight 6 flags 4000
nov 01 18:12:45 sauron tvheadend[3756]: mpegts: test iptv - removing mux prova.m3u - ZDF in test iptv from scan queue

#41 Updated by Jaroslav Kysela almost 2 years ago

Something is wrong. The PAT table is mandatory - do you see any incoming bandwidth during scan ?

#42 Updated by Mario D almost 2 years ago

During a force scan on the ZDF mux, I can't see any bandwidth activities on status->subscriptions. But if I do 'curl http://url.mux' I can see data.
Channels on http://pastebin.com/V4YiW8Zv are publicly available: you could test them if you like.

#43 Updated by Jaroslav Kysela almost 2 years ago

I did a check and I see the problem. The provider uses hls/m3u to serve TS segments of 10 seconds each. I'm sorry, but this is not supported. Each stream has only one PAT packet and TVH requires two at minimum to mark the PAT table as good and also, TVH actually expects continuous TS stream - the code does not walk through all m3u segments and then it should probably refetch new m3u list to continue.

Also available in: Atom PDF