VDR Portal

Hi,

I'm trying to find a sophisticated approach for the generation of unique IDs that I have tried to describe here:
[Announce] Entwurf: Channelpedia generiert automatisch unique Kanal-IDs

I'm happy to translate that into English if needed.

Cheers,
hepi

Quoted from "ludi"

I hope to have been helpful, but there is one problem left: what to do for the channels that are not available in list?

If you write an grabber and your epgsource provides channels that not in that *) list, then you need add the channels to that list (its wiki, everyone can do this). Its forbidden **) to develop an grabber that uses channel ids that are not in this list.

Sure, the German wiki is not ideal, but until someone find a better solution... hepi started a topic about that [Announce] Entwurf: Channelpedia generiert automatisch unique Kanal-IDs

BTW @ludi: Thanks for explain this fine in English. Nice to have an link to point to.

cu

*) http://vdr-wiki.de/wiki/index.php/Xmltv2vdr-plugin
**) not really but the idea are that all grabber use only ids that are in this list.

The question came up how unique channel ids (used by xmltv2vdr or other plugins/addons) should be crafted:

I'm of the opinion that an automated (coded) approach to generate those ids has a few substantial advantages compared to hand-crafting those ids (without using tight rules). An automated approach tries to harvest as many useful meta information as possible from channel list files and tries to create a little database out of that.

Until now, the list of channel id's for the xmltv2vdr plugin was manually extended whenever somebody had written a new grabber. But why should we not instead generate id's for all those channels in the channels.conf straight away and in advance (only excluding unimportant junk channels). We can also use these id's for channel logo assignments. For logo assignments, we need ID's in the same way we need them for xmltv2vdr.

If we are able to parse a channels.conf file line by line and get nice unique ids based on the channel names in the channels.conf file, we also are able to assign the sid-nid-tid of this channel to the unique id. As with different DVB providers those sid-nid-tid's may vary, we are still able to collect all the sid-nid-tid's that match the channel. But this can only work if we take many different channels.confs from many different providers into consideration. Therefore only a channel list directory like Channelpedia has all the data at hand to do it.

So we are not only generating strings based on channel names from the channels.conf (relatively useless on their own), but we store them together with the related sid-nid-tid's so that we collect data to safely find the right unique id for any channel string from a channels.conf. This is useful knowledge later on.

During the process of the id generation we should use one function that will modify the original channel name string according to our needs:

• The function enforces lowercase to avoid clashes with mixed case (KIKA, KiKa, Kika)
  
• The function should for example delete some special characters that we don't want to be part of an id - including blanks (zdf.info, zdfinfo, zdf-info). Stripping special chars can solve many matching problems that would arise if we don't strip them.
  
• The function should add meta information whether the channel is a TV channel or a radio channel (because namesakes may exist).
  
• And the function should assign a language based or region based identifier that claryfies the main audience that the channel is broadcasted to (the main audio language of the channel is helpful to determine this, also the manually created grouping rules in Channelpedia help here to a great extend).
  
• The function should also have access to a list of channel name variants to know that the German channel "Pro7" can also be called "Prosieben" (those alias lists indeed need to be maintained manually. But they only need to contain stuff that isn't solved by taking out the special characters and forcing lowercase).
  
• Inter-channel-relations:
  
  • Channels that exist in a SDTV and HDTV version should get different IDs on one hand, but those id's need to have some kind of relation to each other so that the relation can be seen by software using this data.
    
  • The IDs of regional channel variants (WDR Hintertupfingen, WDR Unterammergau) should also have some kind of relation to each other.
    
  • Timeshift-channels (+1, +24) should have an relation to their "parent" channels.
    
  
  
• The ID strings should in the end still be human readable although grammar rules and word separation are neglected. But even if they are human readable, it might not be easy or sensible to learn them by heart. It is not necessary to learn them by heart or to remember them correctly because humans won't need to deal with them.
  

Most of what I describe above I have tried to do with the unique-id generation demo in Channelpedia. (http://channelpedia.yavdr.com/gen/de_uniqueIDs.html)

When we try to find an existing channel id that matches a channel we can now follow different strategies depending if we only know a channel name or if we also know all the channel parameters in the channels.conf (sid-nid-tid, audio languages, type radio / type TV). In the first case, we would need to do a name based matching. This means we need a function that converts the name given in the same way we have created the channel id from a channels.conf and then look for string matches. In the second case, we can look up the sid-nid-tid is in our database and will find out the correct unique id this way.

For each of these theoretical thoughts there already exists practical source code in Channelpedia that I can link to if you are interested.
The current source code of the unique id demo in channelpedia can be found here:
https://github.com/yavdr/channelpedia/bl…s/uniqueIDs.php

This stuff is not only relevant for xmltv2vdr, but also for example for automated channel logo assignment.

Cheers,
hepi

Quoted from "ludi"

So, if I get it right, an id for a channel in the channelpedia consists mainly of the channel name, a region or country code and maybe some other metadata about whether it is a radio or tv channel.

As much information as necessary to make sure it is a unique string and at the same time human readable and human understandable, meaning: If a human happens to look at the id's what is not mandatory he will still be able to grasp what channel is meant.

Quoted from "ludi"

Instead of continuing now with two separate lists (the unique ids in the xmltv2vdr wiki and those of the channelpedia), I would suggest the following:

hepi: you add an interface to the channelpedia, where the grabber developer can get the ids to use in the xmltv file or xmltv stream of the grabber he is writing. To get the id, he could for example enter data like the channel name, the region where it is broadcasted etc

The thing is that using a strict naming convention is one thing and using Channelpedia data is another thing. If we could set up a naming convention based on the bullet list above people can even create unique channel ids for channels that are not in Channelpedia.

Quoted from "ludi"

Joe_D: you remove the channelnames from the xmltv2vdr wiki and tell people to use the names provided by channelpedia, ideally by providing a link to the interface to search for the unique channel names.

Remains the problem about what to do with the binding channelnames of xmltv2vdr that already exist. One approach would be to change xmltv2vdr to recognize the new and old names. Another approach might be to have channelpedia provide these already existing names. Maybe there is also a third, cuter approach.

Honestly, the sooner an interface to get unique channelnames appears in channelpedia, the better it will be for xmltv2vdr, the developers working on the logos, and probably other people. Anyway, there surely are problems that will only appear once the channelpedia gets really used and I expect that it will have to go through various versions until it offers the various needed features.

I don't think Joe_D should remove anything from the wiki at this time. On one hand I like my approach to solve the id topic, but on the other hand I don't see why Joe_D's approach has to be stopped urgently: I guess it just works and does what it's supposed to do for now.

I don't want to put pressure on anybody and also not on myself. Like I said, my unique id stuff has "draft" status an I was asking for comments in the other thread. I'm convinced that Channelpedia can do a good job in providing id's but there's still work to be done and the best thing would be if at some stage some people will start using it because they feel it helps them. If it's only three to four people discussing this topic, it just is not important enough. It's just that you have started a thread regarding channel names and I dumped all my current ideas that went through my head while working on it.

Regarding xmltv2vdr current ids: I guess there could be a way to keep it compatible with both id systems, couldn't there?

Problems will arise. For the time being I'm only creating IDs for channels from de, at, ch. Until this is all sorted out I will not extend the id generation.

Channelpedia gets "really" used all the time while we are talking here. But its main use up to now is to provide sorted channel lists of which some are updated regularly and automatically.

Quoted from "ludi"

PS: How do I get my channels.conf into the channelpedia, so that it also contains data of the channels that I am using?

You tell me what you have to offer. Which DVB providers do you receive? We can only add those that are not yet part of Channelpedia. Have you followed the Channelpedia announce thread where everything is explained in detail?

Quoted from "ludi"

PS2: Another point that might be interesting for the channelpedia is to enhance xmltv2vdr with an option where the user can choose to upload to the channelpedia the matching between ids of the xmltv file and the channels.conf; this would provide data to the channelpedia about what channel of the channels.conf corresponds to what channel of the xmltv file. @Joe_D: you might perhaps talk to hepi about it.

Like I wrote on the other thread: I don't get that last idea.

Cheers,
hepi