11.3. Sound Quality Problems with Digium hardware (draft)
Lots of people seem to have problems with ticks, noise distortion etc, with the POTS interface cards for Asterisk.
Most of these problems are related to IRQ conflicts, shared IRQ's, disk activity etc.
1. Some background information:
Interrupt? IRQ?
An interrupt is a method by which a piece of hardware communicates with the processor. It's called an interrupt, because the device (such as a network card) interrupts the computer to carry out a function such as I received information, do something with it, or i'm ready with this task, give me a new one.
When a device needs the CPU it sends a request along the IRQ to the CPU. When the CPU gets this request it stops everything it is doing (and save in memory where the CPU left for the task it was doing)
It then services the device that sent the request. It will then get the work it was doing from cache/HDD and carry on what it was doing before that interrupt was sent.
An IRQ (Interrupt Request Line) is a number used to describe an interrupt.
A typical PC has 16 IRQ's, (numbered 0 to 15, only 15 can be addressed due to cascading of IRQ2) most are already taken up by onboard devices such as audio card, com ports,....
Back in the old days, the guy assembling the pc had to assign all IRQs manually, nowadays most of the time this assignment is done plug and play.
(Although some more high end mainboards still allow to bind an IRQ to a specific device manually).
Typically, this is how the IRQs on a random system would look like:
0 -> timer [can't be changed]
1 -> keyboard [can't be changed]
2 -> irq9 cascade [can't be changed]
3 -> com 2/4
4 -> com 1/3
5 -> usually free
6 -> floppy disk
7 -> lpt1
8 -> clock [can't be changed]
9 -> usualy free may be labelled as irq 2 due to cascading
10 -> free
11 -> free
12 -> mouse
13 -> co-processor [can't be changed]
14 -> Pri eide adapter could change if system allows disabling of eide adapters built into motherboard, but be sure that whatever host adapter you boot from uses irq 14
15 -> secondary eide adapter
APIC ?
an APIC (Advanced Processor Interrupt Controller) is a way to get past the 16 irq limit.
For each APIC on your mainboard, you have 24 (with most vendors) additional IRQs available
How can i check if i have APIC ?
Do a cat /proc/interrupts
If that shows one or more lines stating APIC you have an APIC capable mainboard and your OS supports APIC.
This is a sample output from a server with APIC and a TE410p:
APIC will make life a lot easier for you, as IRQ sharing is no longer needed.
However some APIC implementations seem broken. (If you have problems with APIC enabled you might want to try without).
This local APIC might only be used for an APIC timer, im unsure.
There is a second thing called IO-APIC which is enabled by default when using SMP, i couldnt find a way to load it without enabling SMP.
Please note that with an SMP kernel, IO-APIC is always enabled.
2. Common problems.
The zaptel card is not detected
By doing lspci, the card should show up as one of the following:
- Communication controller: Xilinx Corporation: Unknown device 0314 (rev
01) for the TE410p / TE405p
- Network controller: Tiger Jet Network Inc. Tiger3XX Modem/ISDN
interfaceNetwork controller: Tiger Jet Network Inc. Tiger3XX
Modem/ISDN interface for the te110p or the tdm400p
- Network controller: Jens Scoenfeld for the te110p or the tdm400p
The server locks up as soon as i do a ztcfg
- Check if you have a video card with shared memory, if you have, try to add an extra video card, or disable the video card in the bios and use ssh to connect to the server.
shared IRQs or no IRQs for the zaptel card
As seen above, every device that needs an interrupt will be assigned its own IRQ if a a free one is available. (Note that not every PCI cards needs an IRQ!)
If no free IRQ is found, IRQ's can also be shared among multiple devices. (This happens automatically.
These shared IRQ's are used to allow more than 4 IRQ using devices to work on 1 PCI bus - when no APIC is available.
For most devices, this might work just fine, but this IRQ sharing adds more latencies to the interrupts, causing audio ticks, pops,... We don't want that!
A standard linux distro comes with a utility lspci, which can be used to list all PCI buses and cards.
With the -v switch, it will show a list of all cards, + the irqs they are using.
Look carefully if the IRQ used by your card is not also being used by some other device.
If you are using APIC, you can also use the -b switch to see what IRQ the card thinks its using. (the -v switch will show you what IRQ the kernel assigned to the card.)
sample output of lspci -v: (on a server with APIC and a TE410p)
03:09.0 Communication controller: Xilinx, Inc.: Unknown device 0314 (rev 01)
Flags: bus master, medium devsel, latency 64, IRQ 27
Memory at fe8e0000 (32-bit, non-prefetchable) [size=128]
sample output of lspci -vb: (on the same server)
03:09.0 Communication controller: Xilinx, Inc.: Unknown device 0314 (rev 01)
Flags: bus master, medium devsel, latency 64, IRQ 9
Memory at fe8e0000 (32-bit, non-prefetchable)
If the card is using the same IRQ as some other device in the output of lspci -v it will probably cause problems as this IRQ sharing adds extra latencies to the interrupt.
Please note that in the example, the IRQ is 27 and the card thinks its on irq 9. No other device should be using IRQ 27 in this case. (Its normal for some other card to think they are using IRQ 9.
Its ok for non-zaptel cards to share an IRQ. (Although networkcards probably wont like it either).
If you use IRQ sharing, be sure that the devices/cards using the shared IRQ's are all PCI or AGP, no isa.
To resolve this:
- have a look at the bios and disable all the devices you dont need such as USB. video card, extra networkcard to free up some more irqs.
- If that didnt help, see if you can assign an IRQ to a specific PCI slot in the BIOS. (Most modern bioses dont allow this).
- If all of the above didnt help, try moving the card to a different PCI slot, disable the plug 'n play OS option in the BIOS, reboot and try again.
- Enable APIC in the kernel (its enabled on SMP by default, but a command line option might be given to override it.)
- Disable IRQ load balancing in the kernel
The zaptel card is not taking any interrupts
By doing a cat /proc/interrupts ; sleep 10; cat /proc/interrupts
In this case, the t4xxp card is sending interrupts to CPU0, if there would be only zeros in all colums, no interrupts would have been taken since the server was started.
If we do 3854973819 minus 3854963814, we can see how many interrupts the device took in a 10 second period, in this case: 10005 interrupts.
We know that if the card is working correctly, it should take 1000 interrupts per second, as this way of checking is not for an exact 10 seconds interval, the 10005 interrupts looks like a perfectly good result.
If the card is not taking interrupts, make sure you did configure zapata.conf and zaptel.conf, and ran ztcfg -v before starting asterisk.
Loading the kernel modules for the card is not enough to make it take interrupts.
The zaptel card is missing interrupts
Since the zaptel cards really need 1000 interrupts per second, if sometimes it cant do it, audio problems will arise.
To check if your card is missing interrupts, go to your zaptel sources and run ./zttest (stop it with ctrl-c)
According to Digium, if all your results are 99.98% or higher you wont have problems.
If you have results under 99.975%, you will probably have problems, if your lowest result is between 99.975% and 99.98% it might or might not work.
You might want to check this while the cpu is a little loaded and while your networkcard is handling a lot of small packets (iperf?) and thus generating intterupts to really simulate real situations.
Im not really convinced this zttest is actually working correctly, there was a thread on the asterisk mailinglists about this.
Another way to test this is to compile the program zttool in your zaptel sources directory.
First, uncomment the line for the zaptel watchdog in zconfig.h
* Uncomment to enable "watchdog" to monitor if interfaces
* stop taking interrupts or otherwise misbehave
*/
/* #define CONFIG_ZAPTEL_WATCHDOG */
Be sure to comment this line again and recompile and reload the zaptel modules, this setting is not considered stable for production servers.
Doing a Make in this zaptel source directory should compile this program if you have the libnewt libraries installed.
after the zaptel is loaded for some time already run the program zttool in the zaptel sources directory:
Select the pri, and look at the IRQ misses field:
3. General Recommendations / tricks:
Pick an IRQ with a higher priority (server without APIC)
If you get too many IRQ requests (The Digium hardware uses 1000 interrupts per second per card) or u drop the latency you will increase the chance that the CPU wont have time to process your data before it is finished servicing the device that sent the interrupt request.
What happens if the CPU gets two interrupts at the same time ?
The CPU will answer them in an order of priority, based on the IRQ of the device requesting the interrupt.
[0 1 2 8 9 10 11 12 13 14 15 3 4 5 6 7]
Left is higher priority, right is lower priority.
(Thus IRQ 0 will be serviced first, 7 the last).
--> Try to force the card to take an IRQ more to the left, you might be able by changing the pci slot and pray or sometimes by setting in the Bios which IRQ to use.
Pick an IRQ with a higher priority (server with APIC)
The situation with APIC is similar to the situation without APIC.
To see the priority used, first make sure you have enabled APIC in your kernel, then have a look at
/var/log/dmesg
You should fine one or more lines with (one line per APIC on your MB): IRQ redirection table
In this table, the first column stands for the irq in hex, the last for the priority in hex. This is different for every server. In our case we see that irq 23 (hex 17) has the highest priority.
The higher the hex value in the last column, the higher the priority for the device with the irq in the first colum on that same line.
Even with a higher vector (priority) for your zaptel card, that won't help you the least if an IDE interrupt occurs before the zaptel card interrupt, and does some work for several milliseconds, and disables _all other_ interrupts during this time. Priorities don't always help you because the handler for a lower-priority interrupt can still block all higher-priority interrupt handlers.
The vectors should be statically assigned to a PCI slot, so you might need to change the PCI slot of you card to obtain a higher priority.
Check the pci latencies for other devices
When you do lspci -v
You will see an output like this:
02:01.0 Communication controller: Xilinx, Inc.: Unknown device 0314 (rev 01)
Flags: bus master, medium devsel, latency 32, IRQ 24
Memory at fc200000 (32-bit, non-prefetchable) [size=128]
03:01.0 RAID bus controller: Intel Corp. RAID Controller
Subsystem: Intel Corp.: Unknown device 01db
Flags: bus master, fast Back2Back, 66Mhz, slow devsel, latency 32, IRQ 48
Memory at fe200000 (32-bit, prefetchable) [size=16K]
Expansion ROM at <unassigned> [disabled] [size=32K]
Capabilities: [80] Power Management version 2
[...snip...]
For every device, a latency is defined (minimum = 0, maximum =248) the lower this value, the more this device will be willing to give up the bus if another device needs to transmit.
If a device has a setting of 248, it will continue to use the bus for a longer period of time before stopping, while the other device waits for its turn.
You might want to lower the values for all the devices, except for the networkcard and the zaptel cards.
I didnt try any of these settings myself, if you test this, and notice good results, please send us your configuration.
Are you using an IDE drive or doing a lot of HD activity on the same server?
When doing hard disk activity, especially with IDE, the HD controller might use interrupts too long, causing the zaptel card not to be able to do an interrupt 1000 times a second.
You can check if this is the case for your system by simulating some harddisk activity with hdparm
Digium recommends :
"If you are running an IDE hard drive please verify that you are using DMA mode with a UDMA setting of no lower than 2 or higher than 3. UDMA mode 2 is ATA33. UDMA mode 3 is ATA44. This can be done using hdparm.
We suggest using "hdparm -d 1 -X udma2 -c 3 /dev/[IDE Device]". You can check the status using "hdparm /dev/[IDE Device]" and "hdparm -i /dev/[IDE Device]". If you make modifications to your IDE hard drive settings they will only be kept until you reboot. "
Be sure to try kernel 2.6 too, it has better I/O performance than 2.4 kernels.
Choosing a different filesystem might also be a good idea, reiserFS is recommended.
Put your networkcard and pri card on a different PCI-bus
Some mainboards have multiple PCI buses. (like for example the Intel SE7501CW2
As you can see on the image (c) intel - source
This mainbord has 3 PCI buses on two bridges,
quote:"
P64H2: PCI-X 64bit Hub 2.0 I/O Bridge. The P64H2 provides the interface for two 64-bit, 133-MHz Rev. 2.2 compliant PCI-X buses (implemented on Intel® Server Board SE7501CW2 as one bus with one 64-bit, 133MHz slot and one bus with two 64-bit, 100MHz slots). The P64H2 is both master and target on both PCI-X buses.
ICH3-S: IO Control Hub South Bridge. The ICH3-S controller has several components. It provides the interface for a 32-bit, 33-MHz Rev. 2.2-compliant PCI bus.
The ICH3-S can be both a master and a target on that PCI bus. The ICH3-S also includes a USB controller and an IDE controller."
The video card and the two network cards are already on 1 of those buses, there are two free PCI slots on the same pci bus, and 3 more slots on the two other PCI-buses.
In this case, its best to put the card on one of those 3 slots on the free PCI-buses, to avoid all interference and to keep all the bus bandwidth available to the zaptel cards.
Put your networkcard and pri card on a different CPU
Doing a simple cat /proc/interrupts will show you what IRQ is linked to what cpu.:
This screenshots shows the situation where the zaptel card is linked to all the cpu's.
Remember that when changing the CPU affinity for a zaptel card, the numbers won't go to zero in all the columns for the cpus its not using. (they will just stop increasing until you reboot).
How you would do this, will depend on the amount of virtual CPU's available.
It would be a good idea to give the zaptel card its own or multiple CPUs and maybe the network card also its own, and put the disk on the last one (if you have 4).
X-windows and Frame buffers
Asterisk doesnt seem to like X-windows or framebuffers.
- Don't use X on your asterisk server.!
-Disable the framebuffer.
(If you are using a frame buffer, you will probably see a logo on the top lines when working in the console.
To disable it, you need to recompile your kernel (or pass an option to the kernel image when booting).
Use kernel 2.6
Thanks to Ingo Molnar, a lot of changes were made into kernel 2.6 with i/o schedulers.
Kernel 2.6 does a better job than 2.4, you might want to go to kernel.org and download and compile a new kernel.
A comparison between different kernels can be found here
Its a comparison for audio on linux, but the problem they are facing is very similar to the problem with pots hardware.
Please note that that pdf is from 2003, the 2.6 kernel became a lot more mature since then.
People on the asterisk mailinglists said earlier to disable the preemptive kernel in regards to kernel panics, i dont think that still applies.
(im running preemptive kernel myself and am very happy about it.)
On some systems, MMX support in zaptel can be causing clicks in Music On Hold (MoH), on other systems, enabling MMX in zaptel might even cause random crashes
The watchdog function in zaptel is also know to be not good to keep on a production server. (it might cause kernel panics).
Pglashiltep (sa dot mp dot letbu dot v dot i dot l dot at gmail dot com) 18 January 2022 21:13:27 pharmacie ch annecy metz-tessy traitement rhinite allergique <a href=https://www.librarything.com/profile/DavidLoos>Xenical vente</a> pharmacie leclerc toul pharmacie beauvais calais therapie de couple forum pharmacie touati argenteuil trouver une pharmacie autour de moi .
Williamestig (maxim dot opazdalov at yandex dot com) 07 March 2018 12:43:39 <a href=http://achat-cialis-generique.net/>acheter cialis</a> cialis sans ordonnance
<a href=http://commandercialisgenerique.net/>cialis achat</a> achat cialis en france
<a href=http://acquistarecialisitalia.net/>cialis senza ricetta</a> acquisto cialis
<a href=http://comprar-cialis-generico.net/>comprar cialis</a> comprar cialis madrid
Arup (09hgkq1c at yahoo dot com) 23 December 2015 08:26:43 Four score and seven minutes ago, I read a sweet <a href="http://zncctqpup.com">artceli.</a> Lol thanks
Satyam (wm8kopu6tb at yahoo dot com) 20 December 2015 04:48:45 I really wnetad to write down a simple word so as to express gratitude to you for all the marvelous secrets you are posting on this website. My time consuming internet investigation has finally been rewarded with high-quality facts to share with my visitors. I d assume that we website visitors are quite lucky to dwell in a good website with so many perfect professionals with useful secrets. I feel very happy to have encountered your weblog and look forward to plenty of more awesome moments reading here. Thanks once again for all the details.
asr1r (asr1r dot com at hotmail dot com) 15 June 2008 16:02:29
vahid (vahid at gheymatha dot com) 13 June 2008 12:39:39 [url=http://gheymatha.com/school]آموزش فارکس [/url]
iZaak (jan at fousek dot net) 25 September 2006 13:32:29 I've got simmilar problem as Phil and Mario. I haven't run into the white noise, but I can hear repeatiting tick (approximately every minute) Sometimes it's stronger than other times. Please mail me, if you have any ideas.
LV (so at example dot com) 23 September 2006 07:08:54 Linux 2.6.18 kernel comes with three options for the Preemption Model: Server, Desktop, and Low-Latency Desktop. It would appear the Preemptible Kernel option setting of Low-Latency Desktop would offer the best match for a TDM400P card that is otherwise offering poor zttest results from the Zaptel package, perhaps even when sharing an IRQ.
Perhaps also launching the asterisk process as a realtime process would help? ``chrt -v -r -p 5 asterisk''
Mario Vallieres (mario dot vallieres at globetrotter dot net) 20 September 2006 04:10:29 i am having the same problem, when a connection is made i got a white noise but a little echo, pretty weird and every 1 minute i heard a tick.. but the sound is perfect, what could be the problem !?!?!
Phil (phil at giftsthatthrill dot com dot au) 06 September 2006 12:53:56 I'm getting a lot of noise through the analog, cordless phone when a connection is made. Its more like white noise but a little echo also. I've ruled out IRQs as I have the latest IO-APIC and no add on PCI cards except my Digium TDM400P. I've tried the tx and rx params and placed the Digium on a non shared power supply in the 2.6Ghz Pentium Celeron PC box.