Tax evasion is a problem which has caused the Ghanaian tax system right from the colonial times. While some have blamed the situation on the tax authorities for not living up to expectation with regards to tax administration, others attribute it to the unpatriotic attitude of the taxpayers. It was in the light of these challenging positions that this research was carried out using a survey involving tax authorities and individual taxpayers in Abossey Okai, Accra. The study examined the factors contributing to tax evasion in the informal sector, identified the various ways through which tax evasion is perpetrated and the challenges faced by tax authorities in generating tax revenue from the informal sector. The study made use of convenient sampling technique where both taxpayers and tax authorities were selected. The survey revealed that the informal sector of the Ghanaian economy is largely characterized by activities of sole-proprietors who are involved in retailing, wholesaling as well as repairs. It was also revealed that the dominance of sole-proprietorship business in the informal sector has made it difficult for tax authorities to levy taxes on them. The dominance of this business unit is associated with many characteristics which induces tax evasion. These characteristics include but not limited to business registration, lack of permanent business location and predominance of cash transactions. Tax evasion and for that matter the informal sector has caused a great deal of harm to Ghana through its direct and indirect consequences and if care is not taken will gain deep roots in this country. It is therefore recommended that for the Government of Ghana to meet the needs of the citizenry, it would be appropriate to take a look at the factors responsible for the incidence of high tax evasion in the economy.