A typical problem with mixed signal layout involves digital circuits interfering with highly sensitive analog circuits. The A/D converter or the sensitive microphone in the analog circuit could be susceptible to the noise generated by RF Circuit or the fast switching digital circuit.
The most important principle that needs to be kept in mind is that the return current should form as small loop as possible for both - the analog and the digital. The moment the return current loop of the digital circuit or the rf circuit overlaps with the analog circuit, we have the issue where the noise signal will find its path into the sensitive analog circuit.
So, we must see not only the trace in which high speed signal flows but also pay attention to the return ground through which the the signal flows, usually the layer just beneath it. As an example, a slot in the ground plane that extend the return path can create issues.
Analog Circuit Usually does not create problem for Digital Circuit
I use the word usually because, if the Analog Circuit swing voltage is really very high, it may interfere with very low level digital circuits. As an example high power motor and the power relays could effect the digital circuit that operate on voltages as low as 1.5V. That being said, most analog circuits that operate on low voltage do not usually interfere the digital circuit.
On the other hand, the RF and the high speed digital circuit has a good chance of interfering with the analog circuit. The digital return ground should therefore, not pass through the analog ground plane. This is the primary idea why we have separate digital and analog ground planes.
A typical structure of the split Analog and Digital ground plane looks something like this.
As long as the along traces refer to the analog ground just beneath it and the digital traces, just beneath the digital ground beneath it, we should be fine. Practical PCB routing techniques, first tries to find out the extent of the analog and digital traces and then create separate digital and analog ground planes.
However, there may come instance, where it may not be possible to make the analog and the digital traces refer entirely to its respective ground planes as in the following scheme.
If we join the analog and the digital grounds at a single point, using a say, 0603 resistor as show in figure below, it creates an issue, where the return signal is not able to follow the ground just below it. This is highlighted for signal A in the figure.
A good solution to this problem is to route such traces, through near the junction point as shown in figure below.
Not Splitting the Analog Ground Plane
It is possible to keep a single plane and not split the Analog and the ground plane provided we make sure that return path of the digital signal does not cross the analog area.
To understand why is it possible consider a digital signal that has a ground plane at distance h from it. If x is the from center of the trace, then for x/h =2.70% of the return signal is confined within that distance. If x/h is 5.87% of the return current is confined within the distance x. So the key is, to keep the ground ( or the power plane) really close to the signal ( may be 3 mil to 5 mil) and keep the return path such that the digital return never crosses the analog signal.
The figure below shows this scheme.
The idea is theoretically good, but also leads to some practical implementation issue. For example when updating your layout, it is easy to make errors, since the boundaries of the analog and digital sections are not clearly visible. We must, therefore, be very careful, when implementing this scheme and make a checklist, every time we revise the PCB.