Using shifted function keys in XKB keyboard layout - layout

Question

I'm working on a Khmer keyboard layout and I've run into a problem at the last step of the project.
Khmer has 33 consonant characters, 24 vowel diacritic characters, and another 14 independent vowels characters. On top of this, I have included each character's corresponding International Phonetic Alphabet symbol(s) in my layout at the ALT and SHIFT-ALT levels, so very quickly I'm running out of space.
To alleviate this I was planning to map the 24 vowel diacritics to F1 - F12 and SHIFT F1 - F12 (with the corresponding IPA characters at the ALT and SHIFT/ALT levels) but the SHIFT + FN, ALT + FN and SHIFT/ALT + FN combinations don't register. I've tried the combinations in different text editors, browsers, and the terminal but none work. It seems to be a system wide problem.
I am running Ubuntu 10.04.
Here's the section in my layout that pertains to the FN keys:
key <FK01> { [ 0x10017b6, 0x10017c2, voidsymbol, voidsymbol ] };
key <FK02> { [ 0x10017b7, 0x10017c3, voidsymbol, voidsymbol ] };
key <FK03> { [ 0x10017b8, 0x10017c4, voidsymbol, voidsymbol ] };
key <FK04> { [ 0x10017b9, 0x10017c5, voidsymbol, voidsymbol ] };
key <FK05> { [ 0x10017ba, voidsymbol, voidsymbol, voidsymbol ] };
key <FK06> { [ 0x10017bb, voidsymbol, voidsymbol, voidsymbol ] };
key <FK07> { [ 0x10017bc, voidsymbol, voidsymbol, voidsymbol ] };
key <FK08> { [ 0x10017bd, voidsymbol, voidsymbol, voidsymbol ] };
key <FK09> { [ 0x10017be, voidsymbol, voidsymbol, voidsymbol ] };
key <FK10> { [ 0x10017bf, voidsymbol, voidsymbol, voidsymbol ] };
key <FK11> { [ 0x10017c0, voidsymbol, voidsymbol, voidsymbol ] };
key <FK12> { [ 0x10017c1, voidsymbol, voidsymbol, voidsymbol ] };
As you can see I've only filled out SHIFT F1 - F4 so far, I'll do the rest when or if I get those keys working.
Hopefully some of you can point me in the right direction.

Answer 1

I don't know why the combinations don't register, but as an alternative solution think perhaps of utilising more levels per key, for example using level5 (by including one of the symbol definitions to map some key to the level5 modifier) and defining the key mapping using an eight-level type, such as EIGHT_LEVEL or EIGHT_LEVEL_ALPHABETIC.

Answer 2

My guess is that the default fallback "type" of your FN keys are not including enough keys. E.g, if you have
key <FK01> { type = "TWO_LEVEL", [FirstKey,SecondKey,ThirdKey,...]}
then only FirstKey and SecondKey can be generated. If you have
key <FK01> { type = "CTRL+ALT", [FirstKey,SecondKey,ThirdKey,...]}
then the alt and ctrl and ctrl+alt levels should be recognized, giving you 5 levels, 4 of which you might want to change (you probably don't want to change the ctrl level). Do keep in mind that you probably want to be able to use CTRL+ALT+FN to switch out of X to a terminal in case of emergency, or to at least have some key bound to XF86Switch_VT_1 somewhere in your layout.
You might also want to differentiate between Alt and AltGr, the latter named ISO_Level3_Shift in xkb.
If you want to create and use a ISO_Level5_Switch button somewhere on your keyboard you can use the type "EIGHT_LEVEL" or similar to recognize all combinations of (no modifier),level3 and level5 for a total of 8 levels.
(I have Alt_L bound to my left Alt button, ISO_Level3_Shift bound to my right Alt button, and ISO_Level5_Shift bound to my right "windows" button)
Your keymap will probably need to import some of the key types from elsewhere. In the xkb_types section of your .xkb file, add the line
augment "level5"
to include "LEVEL_EIGHT" and other similar types into your layout.
The files in /usr/share/X11/xkb/types/ should show you which types are available on your system, and which file you need to add to your .xkb file with an augment line to use the type. (They might be in a different directory on your system, since I use Gentoo; type locate level5 in a terminal to find where the directory is for you.) If you feel the need, you can also construct your own type directly into the xkb_types section of your .xkb file, after the augment lines. Look in the files in /usr/share/X11/xkb/types/ to see the syntax of a type definition.
You'll also want to add the line to your xkb_compatibility section, not just the xkb_types section; xkb_types provides the types, xkb_compatibility defines modifier key behavior, so while an augment "level5" in xkb_types will give you access to types like "EIGHT_LEVEL", an augment "level5" in xkb_compatibility will make a key bound to ISO_Level5_Shift actually act as a modifier.
Just to give you an idea, my own xkb_compatibility section looks like this:
xkb_compatibility "complete" {
// "pc" would only set Alt, need "pc98" for NumLock
include "pc98" // Numlock
augment "misc" // Alt,Meta,Super,Hyper ...
augment "ledscroll(group_lock)" // Set scroll lock on group switch
augment "xfree86" // Switch out to terminal
augment "caps" // Simple caps thing
augment "iso9995" // level3,groups
augment "level5" // level5
};
I'm not sure if all .xkb files include these sections or if it's just mine.
IIRC, there's often some GUI way to include level3, level5 and the likes when you set your layout rather than in the layout file itself, if that's more to your liking.
If you want to go deeper, I can recommend something like https://www.charvolant.org/doug/xkb/